Talk Archive: The Unreasonable Effectiveness of the Scientific Method
No slides found on page
The Unreasonable Effectiveness of the Scientific Method
Freyja Domville (she/they)
Misskey/Mastodon: @freyja@gensoukyou.jp.net
Blog: https://freyjadomville.co.uk
Hi, I’m Freyja, these are the places you can reach me at to see what I’m up to or how I’m doing. My pronouns are she/they, and if anyone’s up for helping my partner find a job as a junior software dev in Oxford or along the west coast mainline between Bristol and Reading, then please feel free to reach out and get in contact. I’ll provide more details in the mini-pitch section.
This is a diatribe
Personal opinions/thought processes lie herein
I’m speaking here from personal experience and perspective, shaped by both my upbringing and the paths I’ve taken during my career. I’m not trying to sell you anything, just openly sharing a philosophy others might want to use in their own work and life.
Why is this diatribe worth listening to?
- I have a decade in software development, in teams both large and small 🔟
- My specialisms are in accessibility, UX and performance engineering ⏩♿
- Good communication can (from experience) reduce employee burnout ☠️
Cliff notes, I think it helps with the above. Burnout in particular is much more manageable when you can take a step back and make small changes. I’ve got a decade in software development, I have had a varied amount of experience in both performance and user experience work, alongside some accessibility specialisms!
What is the Scientific Method?
So, what is the Scientific Method, and how can it help you?
👀 Make an observation
This is an initial set of measurements to provide a summary of the status quo.
First, you’ve got to think of something you want to fix, or find out. Make an initial set of observations, replicate/reproduce an issue, and so on.
🤔 Form a hypothesis
Begin to reason about what might need to change
Once you’ve collected that data, you can make a hypothesis about what is or isn’t wrong, and how you’d go about verifying the fix, be that writing unit or integration tests if it’s a “Heisen-bug”, or if you think you can make a change that’d reliably fix an issue.
👩🏼🔬 Make a change
to allow for testing the hypothesis
As it says, make the change, land it in a local or test environment for further review, production or otheriwse. Yes, I said production - we don’t judge when IT doesn’t give you a duplicate copy of live to run performance tests on.
📔 Conduct an experiment
Re-run the measurements from the first step, to replicate behaviours
Once you’ve made the change, make sure it affects your tests, or what have you. Replication is important!
🧮 Analyse the results
Did you actually answer the question? Is there a measurable difference?
Now for analysis! Are you done? Did your load tests give you the results you hoped for? What does the extra info look like?
🔁🦆 Repeat as needed
Often it’s useful to iterate on smaller sets of experiments to reach a goal in the longer term
Finally, repeat as needed, until you’re happy. Remember, the scientific method encourages the use of adorable baby steps. You don’t have to do everything at once. If you have a question, stop, think if you need more info, and ask your friendly product owner, rubber duck, or debugger, as appropriate.
”Freyja, why are you telling us this?”
💬 An imaginary audience member for dramatic effect, 2024
Because this is secretly about understanding the needs of people! 🤸🏻
Well… it’s really about understanding systems in general… 📈
but people most importantly! 🤸🏻
“You don’t have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy.”
💬 Jackie Stewart, according to Martin Thompson’s GOTO Aarhaus 2012 talk on the concept
As F1 world champion Jackie Stewart, reportedly said, “You don’t have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy.” The citation I have is a secondary one because I can’t verify it, but the point still stands. In short, the more you know about a system, the easier it is to maintain, and the better a team can work on fixing and improving things.
What is Mechanical Sympathy? 🦾
A philosophy of understanding how a system works to get the most out of the system’s utility
The important point being that it gives you the options and data you need to give insights on where to focus next, without solely being at the whims of a C-Level executive suite. If you don’t have insights into how things are functioning, how can you know if what you’re doing is appropriate?
Why is Mechanical Sympathy useful in software? 🖥️
- It gives you an understanding on where more complex issues may be
- It gives you focus on where to improve if required to
- It provides a mechanism through which an architecture can scale to everybody’s needs.
The elements which Mechanical Sympathy covers are typically on the infrastructure side. Things like race conditions, performance statistics like tail latencies, and other information, comes together to give you an assessment on how many minutes to midnight your system has before catastrophe or some other issue reduces user retention.
The Scientific Method helps with obtaining Mechanical Sympathy!
So how do we obtain this dream of knowing more about our systems? Hint, it’s in the title.
It helps the Developer Experience
- Debug software more effectively by experimenting with different conditions
- Projects can make better scheduling decisions with a consistent framing
- Less change of feeling overwhelmed by a large task
First and foremost, having tickets broken down into smaller tasks, through splitting tickets, stacked merge requests, etc., allows for an even finer grained level of velocity than sprints might get you. Gaining a feel for this also makes the path forward with something involved much simpler, especially as knowledge is shared between the team, which then leads on to happier and more productive people.
It helps the User Experience
- Systems of the future will need to come to where the user is.
- User feedback can be an input into the process of what to do next
- Performance engineering can become an effective part of the accessibility equation
But there’s a user benefit, too. The systems of the future are becoming more and more user-centric as services improve, and as companies try and grasp on to the next big thing when it comes to automation. That makes a good user experience, and a fast site, part of the equation when it comes to accessible systems. It’s not just about the ARIA, an accessible system is also easy to use for someone without an additional need.
💡Takeaway I: The most important time to save in any process or system is people’s time
- This factors into UX, DX, processes around sprints, project delays, the works.
- There is no free lunch
- If you’re slow/complicated/inaccessible, users will go elsewhere (or build it themselves)
To summarise that first little bit, the lines are blurring between user’s needs and how effective systems are at being effective. Especially as we lean into the realm of chat-bots and GPT, it becomes crucial to navigate through this with a mind on performance and good user discoverability for a particular task.
Action: Evaluate your device testing as part of this
- Example: 75th percentiles ⏲️
- Mobile: Samsung A51 (or a Galaxy S7/8 - ish)
- Desktop: 2 core 4 thread Skylake/Bulldozer, spinning HDDs!
This brings me neatly on to a quick aside. Most of the front-end engineers in the room will have seen this, but it’s worth drawing attention to again - the devices we should be testing are not solely the iPhone and the latest Android devices. If you’re not testing with devices that are 5-6 years old, you are losing over 95% of your market.
So… why not “Biological Sympathy” as well? 💪🏼
- People are just as much a part of these systems as the tech we’re building
- Effective products are effective because they understand the needs of the people involved
I do think that Mechanical Sympathy, whilst useful, does have another side to it. The users of our systems are humans, and always will be when it comes to the end result, even if there’s an automated search process in the middle. At the end of the day, they matter, both as a control for a system in operation, and also as a person that will ultimately consume information in new ways.
A personal anecdote…
- Diagnosed with a developmental co-ordination disorder
- Growing up I was always determined to try something different if something didn’t work the first time.
- Technology has shaped me, and it’ll shape future generations even more.
When I was younger, I was starved of oxygen to the brain. The technologies that arose from the 1990’s onwards have shaped the world I grew up in - allowed me to type my exams from the age of 15, and allowed me the path to succeed in a career I have today. They helped me in trying something different, seeing things from another perspective, not always reaching for the common answer.
Takeaway II: The key here is to let people play
- User feedback will be the number one most effective form of feedback any system can have
- Quite common that even the UX engineer won’t necessarily know how a UI performs in the wild.
- There’s always something new to learn under the covers.
This talk was inspired by a guy who was fresh out of a coding camp, asking for insights on what to focus on as he made the transition into industry. And honestly my main piece of advice is this: Play. Rediscover your inner child when thinking about problems. Take some time to reverse engineer a codebase you might not be familiar with, to see how it ticks. And also encourage others to engage in that play. User focus groups are vital to getting hands-on experience. You don’t truly understand the power of accessibility until you see how a user with visual difficulties uses a screen reader. Give them the tools to work on something properly, and their information processing skills are often faster than those of you depending on your eyes. It’ll seriously make you think twice about messing with focus order. I’m just dipping my toes into the realities of video editing, to preserve my talks, and man, it’s a whole new world. I’m excited, and possibly a little intimidated. My plan is to hold on to that where I can, and push myself. That’s really what the Method is all about.
Opinion: The number one skill that most ineffective devs lack is having the tools to debug.
- An unsquashed version control history,
git bisect, and a good use of a debugger will increase effectiveness. - Most junior devs aren’t taught debugging as a dedicated skill in education, and I think the industry suffers for it
- also learn good engineering practice - keep notes!
- Recommendation: Obsidian.
Before we finish, I think there are a couple of actions that the industry could take as a while to benefit those who are learning and are new to this industry, or how the technical side of things truly works. Developers who have an understanding of their tooling are able to get more insight into how things work, or go “dumpster diving”, like me, to find those really persnickety problems. Whenever I see something going on, my racoon-like mind is always scavenging for more info on a particular problem. But here’s the rub - graduates are only really taught the basics in most academic environments, meaning that there’s still a lot here that could be done.
Opinion: Designers could hugely benefit from learning some basic HTML and CSS
- The current model of responsiveness based on width is outdated
- Developers need to step up and provide the tools to give container queries and similar tools as options to a designer.
- if you’ve not got a masterclass culture, then feel free to get in touch with me afterwards.
There’s another aspect to this though. I think that a set of controlled experiments, with examples that are well documented, demonstrated with hands on pairings between designer and developer, can impart knowledge that can be used to more elaborately cater for a multi-screen world, and provide a flexibility I think the industry is only just catching up with.
Opinion: Self-dev time is good, but…
- …it’s worth thinking about how one discipline can help another.
Everybody being siloed means people don’t share in the things others may benefit from. See a cool library? Share it! see a good write-up? Share that too. And also think about setting up a forum or masterclass structure in your organisation to talk about these things. Having spent the last 18 months at 67 Bricks, where I work, it’s been a wonderful breath of fresh air to learn about both the industry and potentially something new in terms of technology or design.
All of this might seem overwhelming…
- It’s worth making sure you’ve got a decent engineering project manager type role to help with client feedback
- Think about how to write software sustainably (more on my blog, also published with the slides!)
And if it all gets too much, make sure you’ve got someone to orchestrate and prioritise what experiments take place. The worst thing to have in a team is burnout. Sustainable software development practices are essential, and whilst I couldn’t fit that in this talk, my blog has a follow up piece of prose about that in the context of the recent xz-utils vulnerability, that’d I’d be happy to come back at a future date and speak about in person.
In short: The Scientific Method is a tool and philosophy that helps me think about my processes on a daily basis…
It helps with learning new things, and forming new ideas…
It serves me well, and I hope it serves you all well too!
I’m looking forward to your own conclusions - keep me posted of your progress
Freyja Domville (she/they)
Misskey/Mastodon: @freyja@gensoukyou.jp.net
Blog: https://freyjadomville.co.uk (working on comments being linked to my Misskey)\
Thanks for listening! I will not be taking questions. 👋🏼
- YouTube version is coming soon (it’ll be embedded on the blog post once recorded)!
- Look at the blog post or subscribe to my Misskey for slide copies and speaker notes