Last week I was in Mountain View and very much looking forward to SFSE! I'm happy to report that it was packed, and if you missed it, before reading my take on it, you may want to check it out yourself without my biased point of view interfering at http://saucelabs.com/blog/index.php/2012/01/sfse-meetup-video-keeping-selenium-tests-100-blue/
I'm interested in learning more about selenium, well, to be specific, I'm interested in using Web Driver with Python and Ruby to automate tests where it makes sense and basically expand my capabilities in web testing. I believe automated testing is very important, and automated checks should be created, maintained, and relied upon as one aspect of quality by the development team. The whole team. I mean that when you ask who is responsible for quality the whole team says, "We are." When asked, "Who fixes the automated tests when they break?" I also want a clear, "We do and we don't develop new features until they are fixed." Anything less means your automation really isn't working to the level that your team can depend on it, so it doesn't have a bright future at this point, or should be rated "needs improvement" by the team. I've seen great test automation only twice. Both times the team owned it, relied on it, and would literally fix it right then. No shutting off tests. If it isn't sacrilege to turn off broken tests that used to work, the automation just isn't reliable enough to be counted on by your team.
So, with all of this in mind, let me share first the happy. So many things to love about San Fran and Mozilla. The patio over there is WOW. Mozilla's got some bucks. That's all I'm saying! I almost felt bad about drinking their soda for a minute, but I think maybe they can afford even my Diet Coke for a night.
Image may be NSFW.
Clik here to view.
Also, a kind fellow took this shot for me! Volunteered even. That makes for a happy tourista. No, my hair hasn't turned black, it's was just a bit dark outside at this point.
Lest you think I was cranky, I had a blast that day. I've been waiting for a huge merge, and my devs did testing together! Yes! They fixed bugs and swarmed for the first time. This team had NO testers, and now they are testing their own code and I feel so proud of them I could burst because they are showing just what a team can accomplish together, and they care about testing their own work. Then I spent the day finding issues, and I even handed off a feature I coded, so I'm learning and teaching on the job which basically means that my self-esteem is at an all time high. I'm seeing a great friend who I enjoy, the sun is shining. I had a new type of food, even red wine the night before. I've basically been drinking free cool Diet Cokes between lovely sun breaks, and despite being pale I'm not sunburned at all since the sun is so mild in the Bay Area this time of year. I'm expecting to learn more automation and even see some developer testing, which I believe is essential for delivering quality software.
During setup, the meetup hosts had some problems, and I was able to suggest ways to use the seating already there so everyone could see. I felt helpful, competent, happy, and above all, like a part of this group excited to see what new things companies are doing with Selenium. It doesn't matter to me where the testing is happening. I am a fan of all testing. I believe that different companies and contexts can tolerate different amounts of risk, so I don't assume that every single company has a use for a test team.
Now the Sad Part
When the talk started, I had an open mind. I am learning more about presenting for developers and mixed tester/developer crowds. Personally, I felt that the slides weren't attractive and that they looked a bit "retro" like a 1995 slide-deck, but I said that before I was upset at all. Honestly, it's a great point that the content is all that really matters. The speaking was good and the story did flow. People were extremely engaged. At least you can tell from this blog, that despite the lack of beautiful fonts or lovely photography in the slides, that the presentation did invoke some emotion in me.
Image may be NSFW.
Clik here to view.
Large crowd had about 100 people. All polite and friendly.
By the time the speaker was done, I felt totally deflated. The presenter said things to me that felt derogatory to testers and dismissive to testing. I felt this way specifically because of the following statements, which lI'll try to be careful to quote and please verify via the recording to get the tone used.
1. We run 90 minutes of tests so we KNOW that everything is safe.
2. We do not have a QA team and we never want one.
3. It is ALWAYS best to automate every test. (this was said despite admitting vast difficulty with automating some tests)
Image may be NSFW.
Clik here to view.
It isn't only that the speaker said these things, but that many others in the room were nodding with such sycophantic vigor that their heads nearly fell off. As I looked around I envisioned the sinister large framed hipster glasses hiding evil eyes identifying me as a tester, and I felt like I wasn't welcome. The inexperience reeked to me, and I felt as if I was the only person who noticed that the demos AREN'T on a real website.
Then--want to know the secret? How they keep the tests all BLUE? It's this secret we've called resetting the baseline in testing for the past 15 years. They aren't even testing the actual THING. They are just notifying themselves of what has changed. It isn't new. If they had testers they might have found that solution sooner. It's a great way to resolve the false positives that are always a part of the normal complexity when doing automated checks. It is sad to me that developers who are inexperienced at testing are so certain that they have all of the answers that they don't even attempt to get any training in testing from those of us who have been doing testing for many years.
Image may be NSFW.
Clik here to view.![]()
I find these slides retro, and note that code review is for people who don't like each other, pairing is for folks who do, but as a tester, especially as a tester who shines in the areas of integration, collaboration, usability, and finding requirements gaps, my opinion was irrelevant in this case. Perhaps a title change from Test Lead to Overlord of Automated Checks (90 minutes to total perfection code confidence) is in order. Maybe some provisos can be added such as "So long as they happen in Chrome, we thought to validate them, they weren't an issue that needed human judgement, and they were in the subset of important tests that we picked, but other than that we are 100% confident."
Image may be NSFW.
Clik here to view.![]()
I feel pretty awful as a public speaker saying anything negative about any person, and especially a fellow member of the tiny group of females in technology who speak, and I want to be clear that this isn't personal. Not only were the speaking skills very good, but her enthusiasm for automation was infectious in a good way. If I've misunderstood, or misquoted, I apologize in advance and will make edits. I do believe that this is a common view in the Bay area, and just a team approach that was being candidly shared for the benefit of other people. I do appreciate hearing the plain truth of the work that is done by the doers and I know that having a good set of automated tests is hard work that takes technical skill and persistence. It isn't just what was said, but the feeling of such enthusiastic agreement and unquestioning that really emotionally resonated with me. The speaker was an experienced Automated Test Creator also speaking on team philosophy. While she's not likely up for Testing Advocate of the Year award(if such a thing existed), but she may be nominated with a few others for "Most Divisive Talk 2012". I've got many more talks and conferences to attend, so this may not even make the top 5 before the year is out, but I sure hope the "Let's trash all testing done by humans for any reason" trend can make the rounds in the Bay area and the rest of the country can keep doing good teamwork, like the agile teams have been lately. I'm seeing progress in developers and testers working together. I'm seeing good classes in mobile testing. From what I understand, this dismissal of the value of all human testing isn't what is happening in Europe. I think this is common in the start-up community and also in the web space specifically where there is venture capitol. It's less prevalent elsewhere from what I can tell so far.
It isn't that they don't have or want testers that bothers me. Why do you hate testing? Without understanding what you are dismissing why don't developers even ASK for our help to learn testing? I promise, we want you to test. Our numbers are shrinking. I never want to visually compare data again. Nearly all of the testing I do is tool assisted. There is lots of testing going on outside of the bay area. If you look, even in the space of the web, the secret shame of even the large companies is the human testing is STILL being done because it is finding critical issues that the automation still isn't finding. Even the boatloads of cash Google and Microsoft have thrown at automating all testing hasn't solved the issue.
When you aren't writing all new code, what then? When you ship your first catastrophic error and find that the dashboards you have aren't really as good as you thought, will that 90 minutes seem like it was enough? Or, when you simply want a human to test for usability, error handling, or the code you forgot to write or put in which even the best code coverage can't check for, the few of us who still put up with the emotional battering it takes to actually give two craps about testing in this hostile environment will be here to help you. How would you feel if a manager took a look at a code generation tool and because it gave an impressive demo felt it wrote better code than you do because it was more consistent? It couldn't replace your talent and experience. What if despite evidence to the contrary, they insisted it was true, and the room full of people clapped? Why are you still sending it to uTest? Because you've only checked it. If no humans have used it, you really have no idea what you are shipping to the public. That may be an appropriate risk in your business. Unlike you, I'm not one to tell people in other professions what is and is not true in their business.
I'm just thankful that there are places where developers and testers still work together. I'll be found in those places. It was difficult to write this. I'm glad I did, because I'm not sure I could give an overview without getting teary at our Seattle meetup tomorrow.
Oh San Francisco Selenium Meetup, you've got yourselves some major fans. On the way there in the traffic Marlena and I joked that the California state motto is, "Keep up or Get Out!" Well, I'm out. I also feel thankful that I work for a client who's been in business with NO testing team for 20 years. I'm the first tester. We now have a team! I'm working full time and another tester is working part time, and even our developers are learning to test. I'm also coding some features, shockingly enough.
It makes me very happy that now the person who hired me says, "I get it! You don't have to explain testing any more to me. I'm seeing it!" Yes. He can survive without testing, but we can all deliver features that have value to the customer faster and more reliably when we all care about and do testing. We don't have all of the automation we would like, and yes, I'd like more of a balance. I care too much for my profession to wish for anything other than forward progress for the industry. I love using software, not just earning a living. That is a big part of why I love testing. Each person I prevent from having a bad experience with software is a point of pride for me. That is true if I do the testing, or even if I teach one developer of a test he might run himself. If I teach a tester one idea they may not have tested before, I'm as happy as if I'd prevented the bug with my own code or my own test (I use both manual tool assisted tests and automated checks).
It isn't the unintentional innocence that upsets me. It's the willful ignorance combined with marginalizing an entire profession. I have more than one skill. I'm not terrified that testing will die, because if it does, there are many other possibilities. I fear for the users of this software, because I am a heavy user of software. I also worry about who you are going to hire to do this totally boring work that no one wants to do. This repetitive droning work that so many companies keep trying to staff. You don't get it. You don't get why you suck at hiring testers. You don't understand why your SDETS are all moving to be full coders now? No one wants to be treated like an idiot and to be your code monkey boy. Stop treating people like they are less than human and then wondering why it's so hard to hire and retain the test coders with your laundry list. Then again, that's for another blog. I'm just thankful that the trip didn't end on this note, or I may have started to drink shots on the plane ride home to soothe my sore feelings rather than working on my testing ideas. There is a reason I'm working so hard to talk beyond just testing. A story of how easy it is to feel "100% safe" after running 90 minutes of automated tests is seductive. There is nothing that can be said to that. You can't warn. You just let it run it's course. Either there will be pain or there won't. I'm certainly at my limit when it comes to fighting a gorilla with a toothpick. I'm here for those who want to work with me, not to convince those who've got the answers already.
Image may be NSFW.
Clik here to view.![]()
The bridge was still beautiful later. The structure wasn't as visible, but it was still there holding up the cars. The lights were far more shiny when the sun was down. It didn't make the supports less useful. I hope when they were last tested, it wasn't just in the emulator.
So, for me, the Selenium SF meetup was the most depressing night of January. I'm happy I met some nice people, and I don't let the opinion of a few folks lead me to assume that everyone shares the same opinion. I have hope for the future, and I am really optimistic to have a happier report for you about the Seattle meetup, which I'm going to be participating in. I'm still planning to take a class to improve my understanding of testing using Selenium, hopefully from Adam Goucher. By the way, if you think testing isn't important, please ask Sauce Labs what they think of the importance of testing and if they have any QA team or testers. I believe they have an agile team, which includes testing in many forms, including developer testing and other agile testing, including some thoughtful human exploratory tests.
Now, on to February and BEYOND! I hope the happy post from earlier was enough to sustain you through this long and pretty sad post.
I'm interested in learning more about selenium, well, to be specific, I'm interested in using Web Driver with Python and Ruby to automate tests where it makes sense and basically expand my capabilities in web testing. I believe automated testing is very important, and automated checks should be created, maintained, and relied upon as one aspect of quality by the development team. The whole team. I mean that when you ask who is responsible for quality the whole team says, "We are." When asked, "Who fixes the automated tests when they break?" I also want a clear, "We do and we don't develop new features until they are fixed." Anything less means your automation really isn't working to the level that your team can depend on it, so it doesn't have a bright future at this point, or should be rated "needs improvement" by the team. I've seen great test automation only twice. Both times the team owned it, relied on it, and would literally fix it right then. No shutting off tests. If it isn't sacrilege to turn off broken tests that used to work, the automation just isn't reliable enough to be counted on by your team.
So, with all of this in mind, let me share first the happy. So many things to love about San Fran and Mozilla. The patio over there is WOW. Mozilla's got some bucks. That's all I'm saying! I almost felt bad about drinking their soda for a minute, but I think maybe they can afford even my Diet Coke for a night.
Image may be NSFW.
Clik here to view.
Also, a kind fellow took this shot for me! Volunteered even. That makes for a happy tourista. No, my hair hasn't turned black, it's was just a bit dark outside at this point.
Lest you think I was cranky, I had a blast that day. I've been waiting for a huge merge, and my devs did testing together! Yes! They fixed bugs and swarmed for the first time. This team had NO testers, and now they are testing their own code and I feel so proud of them I could burst because they are showing just what a team can accomplish together, and they care about testing their own work. Then I spent the day finding issues, and I even handed off a feature I coded, so I'm learning and teaching on the job which basically means that my self-esteem is at an all time high. I'm seeing a great friend who I enjoy, the sun is shining. I had a new type of food, even red wine the night before. I've basically been drinking free cool Diet Cokes between lovely sun breaks, and despite being pale I'm not sunburned at all since the sun is so mild in the Bay Area this time of year. I'm expecting to learn more automation and even see some developer testing, which I believe is essential for delivering quality software.
During setup, the meetup hosts had some problems, and I was able to suggest ways to use the seating already there so everyone could see. I felt helpful, competent, happy, and above all, like a part of this group excited to see what new things companies are doing with Selenium. It doesn't matter to me where the testing is happening. I am a fan of all testing. I believe that different companies and contexts can tolerate different amounts of risk, so I don't assume that every single company has a use for a test team.
Now the Sad Part
When the talk started, I had an open mind. I am learning more about presenting for developers and mixed tester/developer crowds. Personally, I felt that the slides weren't attractive and that they looked a bit "retro" like a 1995 slide-deck, but I said that before I was upset at all. Honestly, it's a great point that the content is all that really matters. The speaking was good and the story did flow. People were extremely engaged. At least you can tell from this blog, that despite the lack of beautiful fonts or lovely photography in the slides, that the presentation did invoke some emotion in me.
Image may be NSFW.
Clik here to view.
Large crowd had about 100 people. All polite and friendly.
By the time the speaker was done, I felt totally deflated. The presenter said things to me that felt derogatory to testers and dismissive to testing. I felt this way specifically because of the following statements, which lI'll try to be careful to quote and please verify via the recording to get the tone used.
1. We run 90 minutes of tests so we KNOW that everything is safe.
2. We do not have a QA team and we never want one.
3. It is ALWAYS best to automate every test. (this was said despite admitting vast difficulty with automating some tests)
Image may be NSFW.
Clik here to view.
It isn't only that the speaker said these things, but that many others in the room were nodding with such sycophantic vigor that their heads nearly fell off. As I looked around I envisioned the sinister large framed hipster glasses hiding evil eyes identifying me as a tester, and I felt like I wasn't welcome. The inexperience reeked to me, and I felt as if I was the only person who noticed that the demos AREN'T on a real website.
Then--want to know the secret? How they keep the tests all BLUE? It's this secret we've called resetting the baseline in testing for the past 15 years. They aren't even testing the actual THING. They are just notifying themselves of what has changed. It isn't new. If they had testers they might have found that solution sooner. It's a great way to resolve the false positives that are always a part of the normal complexity when doing automated checks. It is sad to me that developers who are inexperienced at testing are so certain that they have all of the answers that they don't even attempt to get any training in testing from those of us who have been doing testing for many years.
Image may be NSFW.
Clik here to view.
I find these slides retro, and note that code review is for people who don't like each other, pairing is for folks who do, but as a tester, especially as a tester who shines in the areas of integration, collaboration, usability, and finding requirements gaps, my opinion was irrelevant in this case. Perhaps a title change from Test Lead to Overlord of Automated Checks (90 minutes to total perfection code confidence) is in order. Maybe some provisos can be added such as "So long as they happen in Chrome, we thought to validate them, they weren't an issue that needed human judgement, and they were in the subset of important tests that we picked, but other than that we are 100% confident."
Image may be NSFW.
Clik here to view.
I feel pretty awful as a public speaker saying anything negative about any person, and especially a fellow member of the tiny group of females in technology who speak, and I want to be clear that this isn't personal. Not only were the speaking skills very good, but her enthusiasm for automation was infectious in a good way. If I've misunderstood, or misquoted, I apologize in advance and will make edits. I do believe that this is a common view in the Bay area, and just a team approach that was being candidly shared for the benefit of other people. I do appreciate hearing the plain truth of the work that is done by the doers and I know that having a good set of automated tests is hard work that takes technical skill and persistence. It isn't just what was said, but the feeling of such enthusiastic agreement and unquestioning that really emotionally resonated with me. The speaker was an experienced Automated Test Creator also speaking on team philosophy. While she's not likely up for Testing Advocate of the Year award(if such a thing existed), but she may be nominated with a few others for "Most Divisive Talk 2012". I've got many more talks and conferences to attend, so this may not even make the top 5 before the year is out, but I sure hope the "Let's trash all testing done by humans for any reason" trend can make the rounds in the Bay area and the rest of the country can keep doing good teamwork, like the agile teams have been lately. I'm seeing progress in developers and testers working together. I'm seeing good classes in mobile testing. From what I understand, this dismissal of the value of all human testing isn't what is happening in Europe. I think this is common in the start-up community and also in the web space specifically where there is venture capitol. It's less prevalent elsewhere from what I can tell so far.
It isn't that they don't have or want testers that bothers me. Why do you hate testing? Without understanding what you are dismissing why don't developers even ASK for our help to learn testing? I promise, we want you to test. Our numbers are shrinking. I never want to visually compare data again. Nearly all of the testing I do is tool assisted. There is lots of testing going on outside of the bay area. If you look, even in the space of the web, the secret shame of even the large companies is the human testing is STILL being done because it is finding critical issues that the automation still isn't finding. Even the boatloads of cash Google and Microsoft have thrown at automating all testing hasn't solved the issue.
When you aren't writing all new code, what then? When you ship your first catastrophic error and find that the dashboards you have aren't really as good as you thought, will that 90 minutes seem like it was enough? Or, when you simply want a human to test for usability, error handling, or the code you forgot to write or put in which even the best code coverage can't check for, the few of us who still put up with the emotional battering it takes to actually give two craps about testing in this hostile environment will be here to help you. How would you feel if a manager took a look at a code generation tool and because it gave an impressive demo felt it wrote better code than you do because it was more consistent? It couldn't replace your talent and experience. What if despite evidence to the contrary, they insisted it was true, and the room full of people clapped? Why are you still sending it to uTest? Because you've only checked it. If no humans have used it, you really have no idea what you are shipping to the public. That may be an appropriate risk in your business. Unlike you, I'm not one to tell people in other professions what is and is not true in their business.
I'm just thankful that there are places where developers and testers still work together. I'll be found in those places. It was difficult to write this. I'm glad I did, because I'm not sure I could give an overview without getting teary at our Seattle meetup tomorrow.
Oh San Francisco Selenium Meetup, you've got yourselves some major fans. On the way there in the traffic Marlena and I joked that the California state motto is, "Keep up or Get Out!" Well, I'm out. I also feel thankful that I work for a client who's been in business with NO testing team for 20 years. I'm the first tester. We now have a team! I'm working full time and another tester is working part time, and even our developers are learning to test. I'm also coding some features, shockingly enough.
It makes me very happy that now the person who hired me says, "I get it! You don't have to explain testing any more to me. I'm seeing it!" Yes. He can survive without testing, but we can all deliver features that have value to the customer faster and more reliably when we all care about and do testing. We don't have all of the automation we would like, and yes, I'd like more of a balance. I care too much for my profession to wish for anything other than forward progress for the industry. I love using software, not just earning a living. That is a big part of why I love testing. Each person I prevent from having a bad experience with software is a point of pride for me. That is true if I do the testing, or even if I teach one developer of a test he might run himself. If I teach a tester one idea they may not have tested before, I'm as happy as if I'd prevented the bug with my own code or my own test (I use both manual tool assisted tests and automated checks).
It isn't the unintentional innocence that upsets me. It's the willful ignorance combined with marginalizing an entire profession. I have more than one skill. I'm not terrified that testing will die, because if it does, there are many other possibilities. I fear for the users of this software, because I am a heavy user of software. I also worry about who you are going to hire to do this totally boring work that no one wants to do. This repetitive droning work that so many companies keep trying to staff. You don't get it. You don't get why you suck at hiring testers. You don't understand why your SDETS are all moving to be full coders now? No one wants to be treated like an idiot and to be your code monkey boy. Stop treating people like they are less than human and then wondering why it's so hard to hire and retain the test coders with your laundry list. Then again, that's for another blog. I'm just thankful that the trip didn't end on this note, or I may have started to drink shots on the plane ride home to soothe my sore feelings rather than working on my testing ideas. There is a reason I'm working so hard to talk beyond just testing. A story of how easy it is to feel "100% safe" after running 90 minutes of automated tests is seductive. There is nothing that can be said to that. You can't warn. You just let it run it's course. Either there will be pain or there won't. I'm certainly at my limit when it comes to fighting a gorilla with a toothpick. I'm here for those who want to work with me, not to convince those who've got the answers already.
Image may be NSFW.
Clik here to view.
The bridge was still beautiful later. The structure wasn't as visible, but it was still there holding up the cars. The lights were far more shiny when the sun was down. It didn't make the supports less useful. I hope when they were last tested, it wasn't just in the emulator.
So, for me, the Selenium SF meetup was the most depressing night of January. I'm happy I met some nice people, and I don't let the opinion of a few folks lead me to assume that everyone shares the same opinion. I have hope for the future, and I am really optimistic to have a happier report for you about the Seattle meetup, which I'm going to be participating in. I'm still planning to take a class to improve my understanding of testing using Selenium, hopefully from Adam Goucher. By the way, if you think testing isn't important, please ask Sauce Labs what they think of the importance of testing and if they have any QA team or testers. I believe they have an agile team, which includes testing in many forms, including developer testing and other agile testing, including some thoughtful human exploratory tests.
Now, on to February and BEYOND! I hope the happy post from earlier was enough to sustain you through this long and pretty sad post.