We have reached the half way point in the user testing phase of the research which began on the 11^th April and will run till the 3^rd May. I just thought I would provide a quick snapshot of how things are shaping up in the museum.  To begin, some numbers; so far we have been in the museum on seven individual days and during that time 39 groups have taken part (=5-6 per day). In total, 125 individuals have taken part in using the app. I have interviewed the majority of the groups.

Issues on the ground

We have had a few logistical and technical issues along the way, so I will highlight three of these before I summarise some of the key parts of the research so far. Bad news first…

Issue 1 — The first significant problem we have encountered was on Sunday 12^th April. Frances and I had to close down testing because the top floor of the museum was closed. This was due to a staff absence. When we found out the museum was partly closed-off, we quickly realised that the app will not function as there is no option to ‘skip’ a room if the app sends you there. The upshot is simply that it is important to check with the museum staff that the museum is operating normally.

Issue 2 — On Sunday 18^th, we came across our first significant technical problem. When I attempted to open the app, a notification popped up saying that the Beta testing period had expired. Fortunately, my quick thinking volunteer came up with a temporary solution (manually changing the date on the iPad). I contacted Laura and she put Mark on the case to fix this which he did. However, this issue remains in play; the app needs to be ‘reset’ each month – something to keep in mind if we do any further user testing after the 3^rd May (e.g. on the 10^th June for our festival of education session).

Issue 3 — The third problem is an ongoing one which seemed to peak on Sunday 18^th – the app/ibeacons occasionally not locating effectively. On the vast majority of occasions, the app is locating the user and it finds them and they get the ‘ping’ when they enter them room that they have been directed to. However, on occasion this doesn’t happen. On Sunday 18^th it just so happened that this problem was experienced by three groups who all then returned the iPad because they couldn’t continue with the app (despite multiple attempts by users to be located by walking around and waving the iPad etc.). This sparked us to have a troubleshooting session in the museum on Monday 20^th April…

Troubleshooting session in the Museum – 20^th April 2015

Initially the general thinking was that the depleting battery life of the iBeacons was causing the performance of the app to become unreliable. However, the troubleshooting session with the Aardman technical team (Mark, Nate, and another colleague), revealed that the batteries were minimally depleted with most showing around 90% power remaining. Good news! Most of the iBeacons were installed around October so they are faring well after around six months – this is a big positive in terms of reducing the level of resources required for ongoing maintenance (should they need it).

The sort of bad news, however, is that the troubleshooting session couldn’t specifically identify why the app is intermittently and seemingly randomly not locating users. We generally acknowledge that it can be temperamental at times, and in essence we have simply to work with it being that way. Nate reminded me that one solution for users is to click the ‘open map’ button if it is not locating them and this should give it a kick to find the user – I am now telling users this when I set them up on the app. I think the reality, for the time being at least, is that the app does require a degree of patience from users. Having said that, to reiterate, the app is working almost perfectly a lot more often than not and technical problems are rarely reported back to me in user feedback.

General news from the user testing

One of the bigger changes we have made since the pilot research was to move the Hidden Museum desk to the main foyer – a good move.  You can also see from the photo that we also have our Hidden Museum banner in place. It is fair to say that this has very much improved our general look and visibility and has also increased interest. Relatedly, we have changed recruitment tactics to include on-the-day recruitment, which means we are now inviting visitors from within the museum on the take part. This decision was taken partly because people were asking to take part and we felt like it was defeating the point if we said no, and because it feels like there is only so much that can be done through twitter and online registration. I would say that the combination of the two recruitment methods (online and on-the-day) is favourable. To note, users that we do recruit on-the-day are still required to provide ID and to fill-out the consent form in order to take part.

Feedback from users

Feedback from users has generally been very positive. It is difficult to summarise the feedback in this short space but I will note a couple of things. Where most users agree is on the question of ‘is there a place for the integration of digital technology like the app into the museum experience?’ – the answer has been a resounding ‘yes’, and users have been very clear that they think the concept of the app is great and should be given support.

Secondly, and this is good news, the app appears to be fulfilling its most basic objective – getting visitors to parts of the museum they don’t usually go to. The response on this has been virtually unanimous; the app is taking visitors to the less visited parts of the museum, and for the most part they are finding the experience of going to those parts of the museum interesting and valuable.

With regards to the more critical comments, they are certainly there but they are quite varied so they are difficult to capture at this stage. I will highlight one, and that is the issue of ‘pace’ – quite simply users are not so keen on how fast they end up going around the museum when using the app. When children are part of a group this issue is especially pronounced as they are often leading the group and responding primarily to the app and the instruction to move on. Initial ideas to overcome this include the option to ‘have another challenge’ in the room, or, perhaps an instruction to ‘explore the room before moving on’. Of course, these issues require fleshing out before they can be addressed.

That’s it for now.

Keep spreading the word for people to get involved in the user testing by signing up in advance or dropping in!  @hiddenmuseum

Darren Roberts