Experience report of Mirandola deployment
What follows is an informal experience report of our deployment in earthquake-hit Mirandola, Emilia-Romagna region, Italy.
What follows is an informal experience report of our deployment in earthquake-hit Mirandola, Emilia-Romagna region, Italy. A PDF of the report includes imagery, and is available here: [ PDF ].
Tuesday July 24 2012
It’s early morning, and the weather doesn’t look anything like the forecast storm and clouds. It’s hot, and promises to get even .. better. Think upper 30s.
We’re all slowly converging on Mirandola, in the Emilia-Romagna region, northern Italy. Last night the team from Rome La Sapienza (Fiora Pirra, Panos Papadakis, Mario Gianni, Matia Pizzoli, Arnab Sinha) and Fraunhofer (Slava Tretyakov) already arrived by car, packed with robots. Two big wooden crates with the NIFTi UGVs, two microcopter UAVs, suitcases tools and cables, and of course the obligatory yellow safety helmets.
Nowhere without your helmet.
Salva and Sergio are driving in from ISA this morning, and I am waiting at Modena main station to drive the final bit to Mirandola, together with the rest of the team.
Mirandola. A city some 20-odd kilometers from Finale Emilia, the epicenter of the series of earthquakes that struck this region in May and June this year. A city with thousands of years of cultural heritage ~ and twenty-two thousand inhabitants of which around fourteen thousand whose homes got severely damaged. Many of them are living in tents, in camps or in gardens or in make-shift camping grounds.
As we’re making our way to Mirandola, the very scale of the damage is slowly becoming clear. Houses, barns, buildings in various states of destruction. Roofs caved in, walls cracked or collapsing. In some places, nothing is left but mountains of rubble. At many places, people have put up signs thanking the Vigili del Fuoco for all their help.
Arriving to Mirandola, we first just have to wait before we can go any further. The inner city is Red Zone, pretty much sealed off by the military. No unauthorized access, and saying that we are here to help the Vigili del Fuoco, upon their request, doesn’t get us much further. The local VdF commander comes, we get introduced, but it is not until the ISA people arrive (the VdF academy in Rome) that we are finally allowed to come in. As in, we are escorted to the site where we will be working for the first couple of days: the Chiesa de San Francesco d’Assisi.
The church in some sense epitomizes the rich cultural history of this region. It dates back to the thirteenth century. It is one of the first Franciscan churches in Italy, and it houses several tombs of the famous Pico family. And, like so many other buildings, it was severely damaged during the earthquakes. Only the façade and some of the walls are still standing. The nave and the eastern aisle are mostly destroyed; the western aisle is still somewhat intact, but structurally highly unstable.
And that is where we come in. The Vigili del Fuoco officially requested our aid in assessing structural damage to buildings in this area, and to cultural artifacts like the tombs in this particular church. Robots can go where it is too dangerous for people to enter ~ not just a simple phrase, but the harsh reality, as just a brief while before our deployment two fire fighters got killed in the area, when a roof collapsed. We hope we can indeed be of help.
We find some tables in a seminary next to the church, and start setting up a make-shift command post. Sergio and Salva have brought a power generator, which is slowly and sonorously providing us with electricity ~ and it is, at the latest, then that you realize how (otherwise) eerily quiet it is. No people, no cats, no dogs. Only stone, sand, quiet buildings, and a couple of people frantically typing away at keyboards to get systems ready to go.
Which, for many complex reasons, they aren’t quite. Bleeding edge system functionality cut the wrong way. Some of us explore the outside of the church, trying to understand its layout, the damage done, and possible ways for the robots to enter the church. With help from the local VdF we find out that the doors leading to the western aisle are not (really) blocked.
Once open, the doors provide us with excellent access, and a first view of what things are like “inside.” Piles of masonry, blocks up to 20-25cm, beams, chairs. Dust. Ceilings which are about to collapse. We aren’t allowed to take a single step inside the church (not that we would want to), not even to move some stones away from the doors (which we would have wanted to, just to make a UAV landing zone a little bigger than 40-by-40cm ...). Good thing is, we can establish a first plan of where we will explore, and identify several important landmarks to navigate by: The pillars, the tombs, “the pile at the end,” “the chairs at the end.” That will make planning the exploration, talking about it, that much easier.
But that’s about all that’s easy at that point. 3G-internet access, bright sunshine and no shade, that all doesn’t quite help, and later in the afternoon we decide to go “home” (the bed & breakfast we’re staying at), use landline-internet and work till things are running.
Type “rosmake --pre-clean,” use yet another layer of mosquito-repellent to avoid being eaten even more by local insects, stay calm while trying to eat pizza hold one size-6 nut with one hand and tighten another similar size bolt with the other putting together the landing gear for the UAV and then finding out you should have first done that before fixing the arms with the motors to the center plate and ... It’s getting awfully close to midnight but then we finally have two UGVs, two UAVs, and the entire system running. We can map in 3D, take high-resolution video from a wide variety of omni- and monocular cameras, control the whole lot using a comprehensive user interface setup, and monitor the underlying infrastructure. Tomorrow can come.
Which of course it does, and pretty soon too.
Wednesday July 25 2012
Nothing gets your day started like Italian coffee. Blue sky, hardly any humidity in the air, crickets are making a racket in the trees around the house, whereas the resident red cat takes it all in and decides it’s not for him, continuing to take things easily and comfortably in a lounge chair on the porch.
Whereas we get into an already warm and quite cramped car to make our way back to Mirandola.
We get our tables back from the seminary, and improvise a roof to shield us from the sun. (Yet another logistical miracle from the VdF hat.) Robots get unloaded, monitors and laptops, networking equipment. A big 2.4 GHz WiFi antenna is set up by the side of the doors to the western aisle, just outside the church, with a big cable back to the router at the command post. Just need to make sure nobody trips over it. So out comes the duct tape (where would we be without it?) until the local VdF commander does things properly and arranges a couple of wooden boards to cover things.
The UGVs are hot, and one of the UAVs is up and ready to go. We have come with “two of everything please” to make sure that, should something go wrong (break, get lost, etc.) we at least have back-up solutions to continue the mission. The UAV is a new prototype, with a high-res LUMIX camera on a tilt-unit attached under the UAV main body. The camera can stream its video live, so back at the command post we decide to attach a pair of Vuzix AR goggles to watch it.
It’s pretty amazing.
After an initial test flight around the square in front of the church, we decide to move in and make a first survey of the western aisle. We need to know what to expect, particularly what the situation is at the end of the aisle. Can we get to the nave, either from between the fourth and the fifth pillar, or between the fifth and the sixth? We don’t know, we cannot see from the doors, nor from the outside of the church.
So in we go.
And within a couple of seconds from taking off we are engulfed by fine dust.
Slava is our dedicated UAV pilot, whereas I (gj) sit back at the command post, enjoying the view through the goggles (and how quickly you intuitively start moving your head to look left or right just wishing that the tilt unit would follow!). It’s one thing to look though, quite another to fly. Without high-contrast dark lenses you can hardly see the UAV, the video stream from the UAV provides a better view but we don’t have that for the pilot. In the end it’s a job for two, donning dust masks and hoping that two pairs of eyes see more than just one.
And it works. After several missions, each lasting only up to 4 to 5 minutes in real time but feeling like a lifetime, we get stunning video material of the western aisle. We step through it, looking at possible pathways, looking at the state of the different tombs inside the church, art which just looks like a blur in the video but which in snapshot mode is crystal clear. Things are damaged, but mostly the structure, less so the art. But flying in the dust, no GPS, a somewhat quirky magnetic compass to aid flight control, all leaves the UAV pilot as well as the UAV safety cmd somewhat shaking. And that’s just the start.
It’s time for the UGV to go in. Earlier in the morning, while testing the ROMA UGV on a rubble pile, the differential magically-unexplainably got stuck. Not good. Driving with one bogey in the air is not the right strategy. But, after some wiggling, loosening and tightening, and whatever other kinds of voodoo Mario performed, the differential equally magically-unexplainably got unstuck again. So off we went.
“We” is really an entire team here. Back at the command post Panos pilots the robot, Mario operates various sensors to provide Panos with an optimal awareness of the situation around the robot, and Arnab monitors the network- and system infrastructure, restarting processes whenever necessary. At the door, Matia provides LOS feedback over radio, though we quickly realize that as soon as the UGV is five or six meters away from the door, the command post has a better idea of what things are like than we-at-the-door. So we bite our nails hoping nothing will happen (against all odds we have not attached a tether to the robot), make videos and photos of what’s going on with our robot.
And “Johnny-5” stays alive. Battery, network, all worked fine, and the robot doesn’t have a (noticible) scratch. Thirty-five minutes later, the robot is back, the team is happy-but-fairly-spent. The result is a first batch of camera- and laser data, something in the order of 12GB (“rosbag”), to provide a first view-from-the-ground up until the fourth pillar.
For the moment, that’s enough.
Next we go back in with the UAV, to make recordings of the ceiling with a top-mounted-ceilingwards-pointed camera, and while we’re at it also record more of the tombs. Followed by another mission with the UGV, now driving all the way until the end of the western aisle. It turns out that we cannot get to the nave between the fourth and the fifth pillar, as there is too much rubble there, with a fair amount of long wooden spars and beams blocking our way. We decide to slowly move further, and try between the last pair of pillars. From the UAV video it seemed we should be able to pass there, so that we can go and take a closer look at the stone altar at the top of the nave.
But things are never as they seem.
To get through the opening, it turns out we need to cross the rubble pile at the end of the aisle. Not impossible, in principle. And we do get up the rubble pile. Turning towards the nave proves difficult though: Masonry, wood, the UGV’s threads get blocked. We decide not to risk it. We have video data alright, and as an alternative, we can fly the UAV in from the outside of the church.
That’s easier said than done, though it does turn into a highlight-of-sorts for that day. We need to fly the UAV in across what is left of the outside eastern wall. To see the UAV however we would need to be higher than that. Standing on top of the rubble would be far too dangerous, as would be hanging from the construction holding up the church’s façade. In the end, we drive a fire truck close to the wall, climb onto it, and fly from there.
Spectacular views guaranteed.
Thursday July 26 2012
On one of the walls of the Duomo in Mirandola there is a stone, marking when it was built: 1449. Another landmark, another piece of a long history.
But also here the earthquake left nothing but devastation. The façade of the Duomo has come down, the roof has caved. Walking around the Duomo leaves one with the impression that little of the outer walls is still standing. The bell tower is, still.
There is no way of entering the Duomo through the main doors: They are all blocked, inside and outside. The local VdF commander manages to get in touch with the Duomo’s padre, and obtains permission for us to enter through the administrative buildings attached to the back of the cathedral. The padre arrives on his Holland-style bike, let’s us in.
The administrative buildings escaped unscathed. But that is as far as. As we get to the cathedral, the sight is depressingly familiar. The nave is fully destroyed, as is the western aisle. The eastern aisle, despite appearances when looking from the outside, is still intact. In fact, the floor has been swept ~ we don’t know by whom. Somebody very trusting though.
We decide on a procedure pretty much identical to the one we followed at the Ch. San Francesco: Go in with the UAV, establish a first overview, and make recordings of structural damage to the ceilings, then use the UGV to get more video and create a 3D reconstruction.
After a final flight of the UAV with a 3D (Kinect-style) camera in the Ch. San Francesco (result: 21GB in about 2 minutes...) we take our kit and go to the cathedral. More people have arrived from ISA, including the head of ISA, and by now we have quite the entourage when deploying. Interest is growing, especially among the local VdF: They see what we can do, value it, and request more.
And so, after several very dusty flights inside the eastern aisle and the nave, we fly outside, “up” along the bell tower. The local Vigile del Fuoco needs to know about damage, particularly in the upper part of the tower. But they cannot get there by ladder. Enter the UAV. We exchange dust for heat, enjoy a few of the plums from the garden, off we go. Highly appreciated video.
Together with the padre and the local fire fighters we plan further missions, for the UGV. There are several large cultural artifacts in the cathedral, which we need to have a look at. Including a painting in a room at the top of the western aisle ~ a place we cannot possibly reach with the UAV, as the location itself is beyond line-of-sight and might be obstructed by debris fallen from the roof.
The eastern aisle is fairly easy. We are even, just for a moment, tempted to let the UGV explore the area autonomously. But, only momentarily. Instead of exercising the flexible morphology to cross difficult terrain, we use it to get the robot into more-or-less optimal viewing positions, to record damage to large artifacts ~ pieces of art well over five meters high, from the ground up.
Until we get to go to the room at the end of the western aisle, that is. The UGV needs to traverse a fair amount of rubble behind the altar, at the top of the nave, to get to the room. From then on, the UGV is beyond LOS of the in-field observer.
There are times, when time just feels different. It goes slower, it goes faster. We already experienced that when flying the UAV: What appeared to be a lifetime was in reality just a few minutes.
It is no different here.
We watch the robot drive off, from a door in the administrative building. Into the nave, under a roof which looks like it could collapse even further any minute. Fallen masonry all over, the robot slowly creeping across it, never too fast, never slipping, never doubting it would get to the other side.
And then to disappear from view.
Over the radio we can hear what was going on. Running back and forth between the “observation site” (i.e. the door) and the command post down the hall, each of us can see what it was like. Bright sunlight, masonry, beams, difficult-to-see stairs down towards the aisle, the wall painting we had been asked to record. And very, very tight spaces to move in.
In which the robot almost inevitably gets stuck. There is this thing about understanding the situation around the robot, as a pilot. Something which video can help you establish, but actually, sound, audio, turns out to be really useful. Hearing the robot scrape against something. Making noises it shouldn’t. Listening is sometimes the form of seeing.
But we hadn’t put a microphone on the robot.
Going back and forth, using every available camera on the robot to look at whatever view might be helpful, the UGV team manages to get out, escape through the door, and make a safe return.
A great way to end the day. And, with that, this deployment.
A deployment like this is teamwork. It isn’t just about the robots, and to what extent they are (or are not) autonomous. It’s as much about the people. People determine, often on the spot, what information to gather, where to go, how to continue. People take care of the all-important logistics, local arrangements, down to the simple things such as water, electricity. Coffee.
And all of that we need to fold back into the way we conceive of human-robot teams. We’re learning more and more about how humans and robots can work together in complex deployments like these: The different roles of UAV teams and UGV teams, how we build up situation awareness through a mixture of human- and robot-based observations and communication, what is stressful and how we could alleviate it.
In other words, how we can continue to work on achieving the NIFTi project objectives ;-)
-- Geert-Jan Kruijff (firstname.lastname@example.org)