Mystery - vanishing baskets...

OK, ran into a weird glitch last week, a client brought it to my attention that he had paid, but his payment wasn't registered in our site.
I checked, and in a 24 hours period, we had 12 baskets made, all twelve had successful payments ( checked this in paypal) but five of the twelve baskets were still hanging 'Not Initiated' in CBsubs. Not wanting to annoy our long-suffering customers, i went into their yearly subscriptions pages and updated them manually.

I later noticed that the baskets had vanished entirely.

I had a couple more of these 'glitchy" paypal payments ( confirmed in paypal, but didn't get confirmed to our system) and made a paypal bug ticket on Thursday. (reporting we were seeing only intermittent payment confirmations)

FINALLY, five days later, they finally assigned a tech, who checked the first of my bad baskets and declared that they saw a 200 response from the confirmation URL.

I checked our own weblogs, and saw the pingback URL was indeed hit. But it is a POST, and there is no way of telling if it POSTED valid data, or any at all, for that matter. No PHP errors, and nothing in the CBsubs logs, except the initial basket creation.

Now I go to look for the baskets I saw "not initiated" last week and they are not there.

I do notice that a lot of the payment baskets that did confirm have their confirmation date 8 hours or more EARLIER than their 'creation' date. (time zone issue??)


I looked in the baskets table to see if I could find any 'hidden' baskets but it doesn't appear there are any... is there some mechanism that deletes baskets after a certain time? Odd that the baskets table "id" fields are not contiguous, they seem to jump around a few days, and the basket ID skips digits sometimes. (makes me guess that unconfirmed 'hanging' baskets get cleaned??)

Can you shed any light on this?

I have to manually go back through our payments for the last few days, and verify that everyone who actually MADE a payment has had their subscription renewed...

Nothing NEW on our server, and no sign of unusual server load or activity on that day. I'm leaning towards the idea that its a Paypal glitch on their end, but so far they are brushing me off. Our baskets log has 11,000 successful baskets up to now, and we have not done anything to change configuration

I too have been seeing this over the last week(s).

I also am experiencing this "glitch" on a paypal payment form from fabrik.
If I look at the logs of the IPN in the paypal developer site the ping back url is not complete.

Since it is happening from multiple systems for me it likely is paypal.

This is a big issue at Paypal.com side.

I have opened a support ticket with them yesterday, and got reply this morning that they escalated that to their engineers.

In details:

Since a few weeks, IPN notifications are stripping query strings (around 10-40% of them) and now all are stripped since 8th March!

So they don't work anymore, and if people are not redirected back to site by Paypal for any reason (completing the payment by PDT), payment is lost!

E.g.:
www.example.com/index.php?option=com_comprofiler&view=pluginclass&plugin=cbpaidsubscriptions&cbpgacctno=1&cbppdtback=3bb66c73d1d6afbb11bcac4e6cc3328f&cbpbasket=101&do=gw&result=notify&cbpid=cbp58c844fac4543217212345&format=raw

becomes:

www.example.com/index.php?option=com_comprofiler

You can see that in their admin side in "More"/"Site Map"/"IPN History".

Many reports on the web show same issue: E.g.

stackoverflow.com/questions/42696341/paypal-removes-the-query-string-values-in-the-notify-url-that-is-used-for-ipn

and here:

stackoverflow.com/questions/42696341/paypal-removes-the-query-string-values-in-the-notify-url-that-is-used-for-ipn

If things don't stabilize very quickly (like today), you should also open a paypal support ticket.

In the mean time, in CBSubs general settings, display, baskets, you can increase the basket timeouts drastically, in the hope that they don't timeout when paypal is back, and that you are able to replay their IPNs (if they can fix the badly formed IPNs!).

Thanks very much!!! That explains why it took them FIVE DAYS to even assign a tech to my ticket, and the tech I finally got was apparently pretty junior ( just generically claiming its probably 'something on our end'). We only do a little bit of paypal business - I can imagine we are way at the back of the line compared with vendors doing 5,000 dollars an hour - or 5,000 dollars a MINUTE.

I'm going to continue to monitor things closely, and manually update as needed. Hope they sort it SOON. Is this the beginning of the zombie Apocalypse??

WOW, I just got a followup, from a different paypal tech...

"Do you have a transaction ID for a basket that has not expired, so I can check the IPN data that was sent?

Also, when you receive an IPN, are you running the cmd=_notify-validate callback to verify the IPN data?"



I don't knwo, Do we run the "cmd=_notify-validate" callback ??



When I log into Paypal backend, it says "The IPN feature is turned off. You must turn on IPN in order to create history data."

But then below it gives me detailed history on some (sucessfully completed) transactions anyway...

Looking in there, it seems like our Notification URL string sent was this (??)


and they also show a long IPN string which looks fine as well...

Just not sure what they are talking about with "cmd=_notify-validate" ... ??

CBSubs verifies IPNs (and PDT). PDT return payments will continue to work fine assuming the PDT triggered, but if it's a recurring subscription or the user didn't return to the site after a PDT payment it's entirely IPN based. The issue is PayPal is breaking the notification URL. For what reason I have no clue and why all of a sudden they're doing this I again have no clue. Beat is currently looking into this as quickly as he can. Beat has already explained exactly what's happening in his reply above (linked below).

www.joomlapolis.com/forum/153-professional-member-support/236236-mystery-vanishing-baskets#292276