Testing remote API classes
Moderator: General Moderators
Testing remote API classes
Always a problematic issue - how do you test classes that deal mostly with external API services? I have some tests for the internal logic, as much as it can be separated from the actual request handling.
It is somewhat similar to testing database, only even slower - how do you test that the requests are properly generated without querying the remote API? anyone has some experience / advice? the requests are cURL requests, with a large URL string in the POSTFIELDS, if it helps.
It is somewhat similar to testing database, only even slower - how do you test that the requests are properly generated without querying the remote API? anyone has some experience / advice? the requests are cURL requests, with a large URL string in the POSTFIELDS, if it helps.
- John Cartwright
- Site Admin
- Posts: 11470
- Joined: Tue Dec 23, 2003 2:10 am
- Location: Toronto
- Contact:
Re: Testing remote API classes
Since you shouldn't be testing the external API itself, I would simply use mock data to simulate the responses from the API and test that.pytrin wrote:Always a problematic issue - how do you test classes that deal mostly with external API services? I have some tests for the internal logic, as much as it can be separated from the actual request handling.
It is somewhat similar to testing database, only even slower - how do you test that the requests are properly generated without querying the remote API? anyone has some experience / advice? the requests are cURL requests, with a large URL string in the POSTFIELDS, if it helps.
However, I get that you want to test the generated requested is built properly.. so what I would end up doing is the classes that perform the API requests should be given a curlRequestHandler class, which can also be mocked to return the expected responses (instead of actual performing the requests).
Re: Testing remote API classes
It doesn't matter if I use a mock or not - I can test the request content independently. I trust the cURL extension enough so that once I pass it there, it should work. The problem is the actual testing of the content - the only thing I can do is string comparison, which is very brittle - and in fact, does not ensure that in other cases it won't break. I considered building a class that tests the parameters similar to what the API would do - but that seems like a lot of work (we're talking about the Paypal API here).However, I get that you want to test the generated requested is built properly.. so what I would end up doing is the classes that perform the API requests should be given a curlRequestHandler class, which can also be mocked to return the expected responses (instead of actual performing the requests).
- John Cartwright
- Site Admin
- Posts: 11470
- Joined: Tue Dec 23, 2003 2:10 am
- Location: Toronto
- Contact:
Re: Testing remote API classes
I didn't mean test the curl request handler, I meant to test the request you will pass to it (a mock) -- which may or may not return a suitable response.pytrin wrote:It doesn't matter if I use a mock or not - I can test the request content independently. I trust the cURL extension enough so that once I pass it there, it should work. The problem is the actual testing of the content - the only thing I can do is string comparison, which is very brittle - and in fact, does not ensure that in other cases it won't break.However, I get that you want to test the generated requested is built properly.. so what I would end up doing is the classes that perform the API requests should be given a curlRequestHandler class, which can also be mocked to return the expected responses (instead of actual performing the requests).
That's what I had in mind. I don't think were on the same page though, what were you previously referring to exactly then? BDD?I considered building a class that tests the parameters similar to what the API would do - but that seems like a lot of work (we're talking about the Paypal API here).
Re: Testing remote API classes
No, I meant string comparison - given a predefined array input, check if the result query string is equal to a known string.That's what I had in mind. I don't think were on the same page though, what were you previously referring to exactly then? BDD?
In the end, I went for the parameter check, which wasn't as much work as I'd thought. I'm testing for parameter existence and type. This is the best I can come up with at the moment.
Re: Testing remote API classes
Can you post what is meant by "parameter check"?
Create methods for returning the things you want to assert, if its too brittle you should be asserting on smaller strings. Remember you're not trying to foresee every possible bug when you write tests, you're trying to uncover "corner cases".
Make sure things like curl are wrapped. Ideally you should have tests that do touch the API, and tests that do not. If I had to pick which one to start coding first, I'd say ones that do not touch the API (and I'm pro touching the DB during test). Database tests can be sped up and if running on localhost should never "cry wolf", unlike ones that depend on the network.
Also, its not like you can call up the guys that run the API and ask them to take their server offline so you can test a fault condition. So that's where it comes in handy to make sure you use an object oriented HTTP client, like Zend_Http. This way you can stub out (not mock out [1]) methods to simulate all the error conditions, and build good error handling.
I'm against stubbing things out wherever its not absolutely necessary, but for an API its absolutely necessary.
[1] - Mocks replace your assertion method and verify an "outbound" call of the system under test, while stubs "inject" fake inputs into the system under test, so a unit test can make a traditional assertion on the system under test's outputs.
Create methods for returning the things you want to assert, if its too brittle you should be asserting on smaller strings. Remember you're not trying to foresee every possible bug when you write tests, you're trying to uncover "corner cases".
Make sure things like curl are wrapped. Ideally you should have tests that do touch the API, and tests that do not. If I had to pick which one to start coding first, I'd say ones that do not touch the API (and I'm pro touching the DB during test). Database tests can be sped up and if running on localhost should never "cry wolf", unlike ones that depend on the network.
Also, its not like you can call up the guys that run the API and ask them to take their server offline so you can test a fault condition. So that's where it comes in handy to make sure you use an object oriented HTTP client, like Zend_Http. This way you can stub out (not mock out [1]) methods to simulate all the error conditions, and build good error handling.
I'm against stubbing things out wherever its not absolutely necessary, but for an API its absolutely necessary.
[1] - Mocks replace your assertion method and verify an "outbound" call of the system under test, while stubs "inject" fake inputs into the system under test, so a unit test can make a traditional assertion on the system under test's outputs.
Last edited by josh on Fri Oct 15, 2010 8:23 pm, edited 1 time in total.
Re: Testing remote API classes
Yes, this is what I ended up doing. I'm testing that all the API methods have the required parameters and where applicable I test for format and type.If I had to pick which one to start coding first, I'd say ones that do not touch the API
That's a really rare fault condition (Paypal's uptime is very high). A much more common fault is API version changes that are not backwards compatible (even though you pass the version #, it's not 100% reliable). Also failure on distinct combinations of parameters (edge cases). I figured when I come across those that I'm not aware of yet, I can add specific tests for it.Also, its not like you can call up the guys that run the API and ask them to take their server offline so you can test a fault condition.
Re: Testing remote API classes
Can you post an example?pytrin wrote:Yes, this is what I ended up doing. I'm testing that all the API methods have the required parameters and where applicable I test for format and type.
Rare is subjective, and implies it *will* occur. I worry about error handling before the business logic now, thanks to the sound advice of jenk (I think). "rare" can become "almost every time" when it has to run in a loop 100,000 times or something.That's a really rare fault condition (Paypal's uptime is very high).
Not possible AFAIK. Once a version is released they should make no more changes to that version of the API.A much more common fault is API version changes that are not backwards compatible
Says who?(even though you pass the version #, it's not 100% reliable)
The reason its good to test the edge cases first is because you are less likely to spend time on it later, honestly you're a lot less likely to revisit an old test case until an actual problem occurs. To me, not writing tests for the edge cases first is like accepting the fact there is going to be "undefined behavior" which defeats part of the point of writing any tests in the first place. Yes you'll have a lot of test code, but you'll have rock solid production code that remains rock solid even when abused, and that code will be uber maintainable.. Also failure on distinct combinations of parameters (edge cases). I figured when I come across those that I'm not aware of yet, I can add specific tests for it.
Sometimes you'll end up with users who find out about these edge cases before you think of them too, which is not so good when that behavior starts changing in every release...
Re: Testing remote API classes
Says me. I've worked with it extensively on several recent projects, and some API versions do not act exactly as the manually specifies they should. A part of the problem is obvious difference between the sandbox behavior and the live server (a known problem with paypal)Says who?
I'm not sure how you're suggesting to write tests to cases that are not known to me.To me, not writing tests for the edge cases first is like accepting the fact there is going to be "undefined behavior"
Re: Testing remote API classes
That's unfortunate, but still doesn't imply the live API is going to change from one day to the next? Why the bleep would they do that? Paypal CEO: "Sales aren't hurting enough so let's randomize the API so no one can query it"pytrin wrote:Says me. I've worked with it extensively on several recent projects, and some API versions do not act exactly as the manually specifies they should. A part of the problem is obvious difference between the sandbox behavior and the live server (a known problem with paypal)
The current behavior of your class under an edge case is the only unknown. You should know about the edge cases before you begin programming. For certain things, there are 10 edge cases to every 1 specification. Other times there are 100 edge cases to 1 specification. You should know what you're dealing with, and what could go wrong (user trying to hack you through inputs, buffer overflows, negative numbers, strings when expecting numbers, network timeouts, network cable is unplugged, etc..). I know APIs validate their inputs really well and return certain error codes. Each error code is another edge case.I'm not sure how you're suggesting to write tests to cases that are not known to me.
For example when parsing 2digit and 4digit years, the edge cases in approximate order they should be tested are:
- a blank string
- a non numeric string
- mixed numbers & strings
-1 digit years
-2 digit years but its "00" (test that we keep it as a string and not integer cast it)
- 3 digit numbers,
- 5 digit numbers
Once these tests cases are out of the way, *then* you should have the happy path:
- 2 digit years
- 4 digit years
I'll write the happy paths first from time to time, but I never consider the component "done" until I have a good amount of edge cases surrounding it. What's going to happen if your routes between paypal and your server go down? Are you going to be showing the users a PHP error? A stack trace? A friendly error message? How do you know for sure?
Another example, I had to make something replace " " or "-" with "[ -]" (regex for dash or space). So anywhere they had a dash or a space, I wanted rows that had either dashes or spaces in those spots. I forgot the edge case for someone using "(" and ")" which are regex characters themselves, so when I introduced regex without testing edge cases, I introduced a regression that definitely changed the behavior of other features.
Code: Select all
$value = preg_replace('#[ -]#','[ -]',$value);New implementation with edge cases covered:
Code: Select all
$value = str_replace(array('-','*'), array('##hyphen##','##dash##'), $value);
$value = preg_quote($value);
$value = str_replace(array('##hyphen##','##dash##'), array('-','*'), $value);
$value = preg_replace('#\*#','.*',$value);
$value = preg_replace('#[ -]#','[ -]',$value);Re: Testing remote API classes
I'm talking about facts here, not speculations. The Paypal API is updating frequently (already at version 63). Some changes are not backwards compatible, so you pass the version number to supposedly prevent your code from breaking, only it doesn't always work as intended (especially on the sandbox). Oh, and I'm sure Paypal's CEO has a lot of input on the API. Cause he doesn't have enough work running the company.That's unfortunate, but still doesn't imply the live API is going to change from one day to the next? Why the bleep would they do that? Paypal CEO: "Sales aren't hurting enough so let's randomize the API so no one can query it"
If there was a way to anticipate everything that could go wrong, bugs would not exist. Unfortunately, that is not the case and users will always find unforeseen ways to break your code. This is what I call "edge cases" and I don't see how you suggest writing tests to something I cannot foresee. Instead of writing 100's of tests for guesses of what might happen, I rather see actual world issues. Thanks for the input though.You should know what you're dealing with, and what could go wrong
Re: Testing remote API classes
I'm not sure I follow, you're saying the new versions are not backwards compatible with the old versions. How would that cause the old versions to stop working? You're saying when they introduce version 63, version 62 changes as well?pytrin wrote:I'm talking about facts here, not speculations. The Paypal API is updating frequently (already at version 63). Some changes are not backwards compatible
I'm not suggesting writing tests for stuff that isn't going to happen, I'm suggesting writing tests for stuff that will happen. If you think paypal's API is infallible and experiences no down time ever at all, then don't test your edge cases... its your app not mineI don't see how you suggest writing tests to something I cannot foresee. Instead of writing 100's of tests for guesses of what might happen, I rather see actual world issues.
Look at all the error codes: https://cms.paypal.com/us/cgi-bin/?cmd= ... errorcodes
Do you need 1 test for each error code? Of course not. But in my experience you'll need one that is a "catch all" so to speak.
Look at the unit tests for Zend_Rest:
Should throw exception if no URI in object
RestFixesPathWithMissingSlashes
testInvalidXmlInClientResultLeadsToException
This last one has 2 issue #s next to it. Missing that obvious edge case cost them face. (doesnt exactly take a genius to forsee that someone could input broken XML, just a little bit of careful planning). Also it takes more time to come back to it later, rather than just build a hardened class from the start.
Another reason its important is defect localization. When a test breaks one day, is it going to tell you what the problem is from reading the name of the failed test?
Well I've said a lot, I recommend reading google because enough cannot be said: http://www.google.com/search?q=testing+ ... 9999a5d8d0
http://brian.pontarelli.com/2006/12/04/ ... e-testing/
Also, if I'm not mistaken in this thread you argued with me against jenk in favor of testing the actual database so you can test more edge cases. I'm wondering why the different opinion now? viewtopic.php?f=39&t=112413&p=592651&hi ... es#p592651
Re: Testing remote API classes
In this case it soundslike the correct way to test is a mock and a string comparison. Is parameter order important? If so write a custom expectation which doesn't care about order.Eran wrote:Always a problematic issue - how do you test classes that deal mostly with external API services?
However, there's not much you can do if paypal won't respect version numbers. This really is not an acceptable way to do business. Even small errors or rough edges in financial transactions can have a huge effect on customers' confidence in your web site as a reputable organisation.
I don't know enough about the specifics but perhaps you could insert some kind of PaypalSanityCheck policy between the paypal response and the rest of your code so you can detect problems and try to handle them gracefully.
Re: Testing remote API classes
I'd still love to know specifics about that part, nothing personal but I won't believe it until I see it. If it were true you'd think we'd hear about it, or Google searches would turn up threads like this or people blogging about it.. Its hard to imagine they'd do something that hinders you making them money. If its true you should post more details. I'd think you'd want to shed light on it.McGruff wrote: if paypal won't respect version numbers. This really is not an acceptable way to do business.
Re: Testing remote API classes
As I mentioned (I'm not sure if you bother to read my posts), the problems are mainly on the sandbox. Unless you want to test using real money, you have to use the sandbox for development. Google for "paypal sandbox issues" and you will find plenty of threads (especially on the paypal forums) of people having trouble with it and Paypal officials fixing their problems manually.
Here are some to get you started - http://www.joeaudette.com/paypal-sandbo ... -work.aspx
http://www.strictlyphp.com/blog/2008/10 ... -problems/
https://www.x.com/thread/46468
Here are some to get you started - http://www.joeaudette.com/paypal-sandbo ... -work.aspx
http://www.strictlyphp.com/blog/2008/10 ... -problems/
https://www.x.com/thread/46468
I hope you understand that you are saying I'm either lying or not knowing what I'm doing. If that's not personal, I'm not sure what isnothing personal but I won't believe it until I see it
Hard to imagine that large corporations have tons of legacy code and bad development practices? not really. They certainly put a lot into support, and their support guys constantly make manual fixes for complaining users.Its hard to imagine they'd do something that hinders you making them money