The digital apprenticeship service is a complex programme with 7 scrum teams developing different elements of the service.
We had already carried out lots of user testing on the individual component parts, but it was really important to do a private beta across the whole service so that we could test how it all fitted together.
How it worked
We wanted all 7 scrum teams to have sight of everything we were learning in private beta, which logistically is quite a challenge.
We set up a wall of research (see photo) which was highly visible to all teams. We printed the screens which were being tested each week and annotated them every day with comments from employers on post-it notes. Comments were either green (it’s fine, I can use it), amber (I’m a bit confused) or red/pink (I can’t carry on). Other colours were used for our notes. This wall became a visual way of sharing what employers were saying (for example, a pile of red post-its on one screen showed there was an issue).
Once a week, the user research clan (user researchers from each scrum) invited all teams to stand in front of the wall and ask questions about employers’ feedback. Following the session, the researchers worked with the relevant product owners to make decisions about changes which were needed to the product.
Flexible research
Everyone had to do their bit to make it work; at times it was a real challenge to get things done on time, and sometimes, in the spirit of agile, we would change the priorities of what was built first. This meant that we had to be flexible about our research and re-think where we could best add value.
We ran the cycle for two months with 100 employers. By the end, we were much closer as product teams and we all understood how the service fits together from an employers’ perspective. We’re all working to a single backlog so that we keep the end-to-end experience in the front of our minds.
Key lessons
A lot of changes we made were in the detail: how to word things better or where to place items on screen. But there were some bigger, more important learnings, too:
1. Details about our users
We learnt more about exactly who our users will be. During the alpha phase, the people in the organisations we’d engaged with who had been keen to talk to us were generally apprentice or training managers, often sitting within HR, and usually at middle management level. When they started to use the service, they quickly realised there were some tasks they couldn’t complete alone. Because of this, we started to engage with other potential users - like members of the payroll team or more senior managers.
2. How different organisations are structured
Again, because we were asking employers to use the service for the first time in a real situation, and input real data, they often needed to communicate with different parts of their organisation. Sometimes, these different departments might be completely separate or even outsourced. Awareness of this really helped us to tighten up our segmentation and personas.
We found out, for example, that finding the right person with access to Government Gateway details was really hard. If you’re a small organisation, those details could be with someone sitting at the next table. If you’re a big organisation, this could be outsourced or could be handled by someone in a completely different department or building.
3. Assumptions about users’ knowledge
During alpha, all of our testing was moderated, and most of it on small parts of the service at a time. This meant that the participant was usually given some context into the service and what was being tested. Unmoderated testing in private beta really challenged our assumptions around users’ understanding of what the service is for and how to use it.
Next steps
Private beta taught us a lot about our users and what they did and didn’t know. I’m looking forward to finding out even more as we enter public beta. You can follow our progress here on the SFA Digital blog by signing up to email updates or our RSS feed.
13 comments
Comment by Glenn Thompson posted on
When we will be able to access the DAS please? It has been published for a while now, that from January 2017 we will be able to open an account on the service, but so far I have not been able to find a way to do this?
Comment by Ann posted on
Hi
I am concerned with setting administrators up on the web site just for the levy, will we be able to restrict them from seeing all the PAYE data
Thanks
ACH
Comment by Sandra Lingwood posted on
Could you tell me if this is all set up for companies to use yet?
Comment by Jemima Gibbons posted on
Hi Sandra,
The service is now open to all levy-paying employers for registration:
https://www.gov.uk/guidance/manage-apprenticeship-funds
Comment by Sandra Lingwood posted on
When logging on do I go on as a new user or existing - I have a log on for the Government Gateway
Thanks
Sandra
Comment by Erika Marsh posted on
Same comment from me. When can I sign up my company to the Digital Apprenticeship Service?
Comment by Carl Sutcliffe posted on
Same question from me, its difficult to get traction on our Levy strategy, when key components are subject to vague deadlines
Comment by Keith Tidmarsh posted on
We are towards the end of January and the published information suggests that levy paying companies will be invited to register on DAS from January 2017 - is this revised or is it to be done incrementally with batches of companies at a time? so maybe not open to all in January?
...Or have I missed the invite?
Comment by HTJ posted on
Does anyone else find it frustrating that we have to search for this? As a business who will immediately be captured by the levy I'm frustrated by the lack of information and constantly having to look for updates.
Comment by Wendy Brown posted on
I Agree. I have been checking on weekly , now daily basis. Very little news about the go live date
Comment by Vanessa Clynes posted on
Thank you for your comments, everyone.
We have invited a small selection of employers to register on the new apprenticeship service over the coming weeks.
We are phasing the registration so that we can continue testing and improving the service, before inviting all levy-paying employers to register by the end of February.
In the meantime, there are a few things you can do in advance so that you're ready to register. To register, you will need:
● Government Gateway login details for the PAYE schemes you want to include in your account
● details for each organisation that will be making an agreement with a training provider for apprenticeship training, including the Companies House or charity number (if you have one)
Government Gateway is the online service that your payroll or finance department use to submit tax returns. If you don’t have your Government Gateway login details, you may need to ask someone from your payroll or finance department to help you set up the account, or to set it up on your behalf.
If you are registering on behalf of multiple connected companies or charities, you need to think about:
● the structure of your organisations and PAYE schemes
● how you are going to spend your funds
You’ll need to register every organisation that will make an agreement with a training provider for apprenticeship training.
When registration is open to all employers, we will ensure that we give you the link here on the SFA Digital blog.
Comment by Chris Wiper posted on
Hi Vanessa,
I was just checking if there had been any updates and saw your comment above. When you say you are phasing the registration how are you planning to contact companies? We employ a payroll management company, I assume they will not get our invite instead of us?
Regards,
Chris
Comment by Anna Andersen posted on
Hi
I'm a user research on a government project about to go into private beta and it would be great if I could get a chance to ask your user research a few questions about your approach.
Anna