Hello folks!

Most of you have probably encountered those annoying 502 Error/Can't Connect pages while navigating the PaperBackSwap site, that started about a month ago.

502 Errors (or "Can't Connect" errors) mean the system is unable to process your request (clicking a link on our site to ask for a page to be displayed = a request). So it was clear that our servers were behaving like they were overloaded.

We have been working hard the whole time to track down the source the instability. For a while it seemed it was just a symptom of generally overburdened servers (our system stores and manipulates a lot of data, every time it serves up a page to a user -- particularly the pages with lists). So we spent a good deal of effort on streamlining the site, purging unneeded stores of data. Those measures did result in transient improvement (which made us think for a while that we were on the right track), but the problem would only get worse again -- as you all noticed!

We're very glad to say we have located the actual source of the problem -- our servers were being hit by "invalid API requests," 3 per second, from three sources (one in Thailand, two in Canada), around the clock. That means that three different computers were relentlessly bombarding our system with queries that would occupy the servers but that could not be processed.

We've blocked them now, and after we've done some tidying up, things should get back to normal. 

The GOOD NEWS

• This was not an attack that aimed to expose or collect any personal data. An API is a way for two software applications to talk to each other -- for example, our system uses an API to interface with the USPS system to get barcodes for Printable Postage, and also scan information for those barcodes.

  • An analogy for this attack would be: you're trying to talk on the phone and your child decides RIGHT THEN to start saying "Mom - Mom - Mom" 3 times a second.  It is hard to carry on a conversation with that going on.

  • It's actually MORE like you are doing complex calculations and your toddler is shouting a foreign language at you, three times a second, and each time he shouts you have to try to figure out if he is saying something important that you need to act on, before you can decide to ignore him and go back to what you were doing.

  • Each time you load a page on our site by clicking a link, the page is created from tables of data in the time (normally a split second) before it shows on your screen.

  • So, our servers were trying to "carry on a conversation/do complex calculations"-- they were trying to compose and deliver a specific page for each specific link that was clicked, simultaneously for all of the PBS members who were clicking links on the site, but the servers were being interrupted three times a second from three different sources with nonsense that had to be evaluated each time before it could be ignored.

• It is IMPORTANT to be very clear that this was not an attack designed to steal data -- it was just designed to mess with us (and it did!).

  • You might ask, Why do some people (mostly teenage boys) like to do this kind of thing?

  • We don't know why-- it may be a function of too much time on one's hands + no responsibility to pay bills + immature/undeveloped moral sense. That's a confluence of circumstances that can lead to significant nonsense.

• So: your data was not stolen. No one was even trying to steal data. Our site is secure. This was not a data breach.

More GOOD NEWS

• The work we have done to streamline the system is going to pay off soon, in system efficiency and speed once we are done tidying up after.

WHAT WILL HAPPEN FROM HERE:

• Things will continue to be less than optimal for the next few days as repairs and preventive measures are taking place (software and hardware getting restarted).
  • We will do the bulk of the work at night, to minimize disruption to the site.
  • This will take a few days. 
  • This work will include measures to prevent this from happening again.
• Then the system will return to normal (meaning, you should see no more 502/Can't connect errors!).

We are very sorry for all the inconvenience. We know it has been very frustrating! We apologize for being so quiet -- there was nothing to report except "We're working on it". We're also sorry it took such a long time to make progress on the problem. You all were really patient about it (yes, even those of you who sent in messages cussing at us! We felt your pain), and we appreciate it.

Thanks for hanging in there! 

The PaperBackSwap Team

Thanks for all the work that went into finding the problem

Thank you.

does not seem to be fixed yet. I'm still getting bad gateway disruptions.

Charles, note that last portion of their post:

WHAT WILL HAPPEN FROM HERE:

• Things will continue to be less than optimal for the next few days as repairs and preventive measures are taking place (software and hardware getting restarted).
  • We will do the bulk of the work at night, to minimize disruption to the site.
  • This will take a few days. 
  • This work will include measures to prevent this from happening again.
• Then the system will return to normal (meaning, you should see no more 502/Can't connect errors!).

our servers were being hit by "invalid API requests," 3 per second, from three sources (one in Thailand, two in Canada), around the clock.

So.... Where are the Navy Seal "hit" teams when we really need them?