How do robots ‘see’ the world?

Disney’s WALL.E needed to see all the rubbish on Earth so it could clean it up. AAP Image/Tracey Nearmy

Jonathan Roberts, Queensland University of Technology

The world has gone mad for robots with articles talking almost every day about the coming of the robot revolution. But is all the hype, excitement and sometimes fear justified? Is the robot revolution really coming?

The answer is probably that in some areas of our lives we will see more robots soon. But realistically, we are not going to see dozens of robots out and about in our streets or wandering around our offices in the very near future.

One of the main reasons is simply that robots do not yet have the ability to really see the world. But before talking about how robots of the future might see, first we should consider what we actually mean by seeing.

I see you

Most of us have two eyes and we use those eyes to collect light that reflects off the objects around us. Our eyes convert that light it into electrical signals that are sent down our optic nerves, which are immediately processed by our brain.

Our brain somehow works out what is around us from all of those electrical impulses and from our experiences. It builds up a representation of the world and we use that to navigate, to help us pick things up, to enable us to see one another’s faces, and to do a million other things we take for granted.

That whole activity, from collecting the light in our eyes, to having an understanding of the world around us, is what is meant by seeing.

Researchers have estimated that up to 50% of our brain is involved in the process of seeing. Nearly all of the world’s animals have eyes and can see in some way. Most of these animals, insects in particular, have far simpler brains than humans and they function well.

This shows that some forms of seeing can be achieved without the massive computer power of our mammal brains. Seeing has clearly been determined to be quite useful by evolution.

Robot vision

It is therefore unsurprising that many robotics researchers predict that if a robot can see, we are likely to actually see a boom in robotics and robots may finally become the helpers of humans that so many people have desired.

Early days: A vacuum cleaner that can ‘see’ where it needs to clean.

How then do we get a robot to see? The first part is straightforward. We use a video camera, just like the one in your smart phone, to collect a constant stream of images. Camera technology for robots is a large research field in itself but for now just think of a standard video camera. We pass those images to a computer and then we have options.

Since the 1970s, robot vision engineers have thought about features in images. These might be lines, or interesting points like corners or certain textures. The engineers write algorithms to find these features and track them from image frame to image frame in the video stream.

This step is essentially reducing the amount of data from the millions of pixels in an image to a few hundred or thousand features.

In the recent past when computing power was limited, this was an essential step in the process. The engineers then think about what the robot is likely to see and what it will need to do. They write software that will recognise patterns in the world to help the robot understand what is around it.

The local environment

The software may create a very basic map of the environment as the robot operates or it may try to match the features that it finds with a library of features that the software is looking for.

In essence the robots are being programmed by a human to see things that a human thinks the robot is going to need to see. There have been many successful examples of this type of robot vision system, but practically no robot that you find today is capable of navigating in the world using vision alone.

Such systems are not yet reliable enough to keep a robot from bumping or falling long enough to give the robot a practical use. The driverless cars that are talked about in the media either use lasers or radar to supplement their vision systems.

In the past five to ten years a new robot vision research community has started to take shape. These researchers have demonstrated systems that are not programmed as such but instead learn how to see.

They have developed robot vision systems whose structure is inspired by how scientists think animals see. That is they use the concept of layers of neurons, just like in an animal brain. The engineers program the structure of the system but they do not develop the algorithm that runs on that system. That is left to the robot to work out for itself.

This technique is known as machine learning and because we now have easy access to significant computer power at a reasonable cost, these techniques are beginning to work! Investment in these technologies is accelerating fast.

The hive mind

The significance of having robots learn is that they can easily share their learning. One robot will not have to learn from scratch like a newborn animal. A new robot can be given the experiences of other robots and can build upon those.

One robot may learn what a cat looks like and transfer that knowledge to thousands of other robots. More significantly, one robot may solve a complex task such as navigating its way around a part of a city and instantly share that with all the other robots.

Equally important is that robots which share experiences may learn together. For example, one thousand robots may each observe a different cat, share that data with one another via the internet and together learn to classify all cats. This is an example of distributed learning.

The fact that robots of the future will be capable of shared and distributed learning has profound implications and is scaring some, while exciting others.

It is quite possible that your credit card transactions are being checked for fraud right now by a data centre self-learning machine. These systems can spot possible fraud that no human could ever detect. A hive mind being used for good.

The real robot revolution

There are numerous applications for robots that can see. It’s hard not to think of a part of our life where such a robot could not help.

The first uses of robots that can see are likely to be in industries that either have labour shortage issues, such as agriculture, or are inherently unattractive to humans and maybe hazardous.

Examples include searching through rubble after disasters, evacuating people from dangerous situations or working in confined and difficult to access spaces.

Applications that require very long period of attention, something humans find hard, will also be ripe to be done by a robot that can see. Our future home-based robot companions will be far more useful if they can see us.

And in an operating theatre near you, it is soon likely that a seeing robot will be assisting surgeons. The robot’s superior vision and super precise and steady arms and hands will allow surgeons to focus on what they are best at – deciding what to do.

Even that decision-making ability may be superseded by a hive mind of robot doctors. The robots will have it all stitched up!

The Conversation

Jonathan Roberts, Professor in Robotics, Queensland University of Technology

This article was originally published on The Conversation. Read the original article.

Star Wars: these could be the droids we’re looking for in real life

BB-8 (left) is a new droid addition to the Star Wars universe. Disney

Jonathan Roberts, Queensland University of Technology

The latest episode of Star Wars is now upon us and has unleashed a new era of science fantasy robots, or “droids” as they are known.

One of the heroes of the new movie The Force Awakens is BB-8, a cute but capable spherical droid that is at the centre of the story (sorry, no spoilers).

But droids have been at the heart of the epic science fantasy saga since the original Star Wars movie back in 1977, when C-3PO uttered the immortal words:

I am C-3PO, human-cyborg relations. And this is my counterpart R2-D2.

Star Wars has always been a droid story, just as much as a story about the Skywalker family.

The old and the new: C-3PO (left), BB-8 (centre) and R2-D2 (right) from the Star Wars universe.
Reuters/Carlo Allegri

Even though we all know that Star Wars happened a long time ago, in a galaxy far, far away, just how good has it been a predicting the usefulness and development of robots on our own planet today?

Is that you R2?

For those non-Star Wars experts reading this, R2-D2 is an R-series astromech droid. Such droids work on spaceships and are particularly good at working outside in the vacuum of space. They are the mechanics of space travel and are packed with tools and know-how to fix things. They also seem to be fully waterproof, can fly short distances using deployable thrusters and somehow possess a cheeky character.

But did you know that working in orbit around Earth right now is NASA’s Robonaut 2, also known as R2. It is one of the International Space Station’s test bed droids, having a humanoid shape and proportions so that it can undertake maintenance tasks originally designed for human astronauts.

Robonaut2 – or R2 for short – from NASA and General Motors, is a robot designed to work side-by-side with people in difficult or dangerous places on Earth and in space.
NASA

Perhaps in the future, when all spaceship maintenance will be performed by droids, this real R2 unit will replace the humanoid form.

The diplomatic droid

The golden humanoid C-3PO is a protocol droid fluent in more than six million forms of communication. A protocol droid’s primary purpose in Star Wars is to help non-droids, creatures of all kinds, communicate with one another and generally avoid potentially dangerous misunderstandings.

If there were protocol droids in Mos Eisley’s Cantina then maybe no-one would have shot first! But as the bartender said of R2-D2 and C-3PO: “We don’t serve their kind here.”

We have human diplomats in our world to negotiate and attempt to head off conflict, and there seems no need for a mechanical interface such as a protocol droid.

But we are seeing translation apps on our phones and their accuracy is improving to the point where it is conceivable that live language translation between two people speaking to one another may not be too far away. Until we find non-human sentient equals then there will be few diplomatic jobs for C-3PO-like droids here on earth.

A place that we are likely to see humanoid robots like C-3PO is as artificial companions and carers. The advantage of a humanoid robot is that it should be able to cope in our homes or care facilities as they have all been designed for humans.

This is one of the great advantages of the humanoid robot form, although there is the so-called “uncanny valley” to deal with and the feeling by some that we should always ensure people have a human touch.

Best Star Wars Droids

A way of thinking about the dozens of droids of Star Wars is to classify them by how they are used. We have seen them being used in applications as diverse as farming, medicine, war, torture and space exploration.

Farming robots

When R2-D2 and C-3PO escape Darth Vader and land in their escape pod on the sand plant of Tatooine, they are picked up by the Jawas scavenging for droids to sell to local moisture farmers. The lack of labour on Tatooine results in droids being critical for the functioning of the farms.

Note to non-Star Wars experts: Darth Vader himself, or the least a young Anakin Skywalker, built C-3PO on Tatooine from spare parts.

In the past year alone, very capable agricultural robots have been demonstrated by Queensland University of Technology, The University of Sydney and by Swarm Farm Robotics.

Robots down on the farm.

Many other research organisations and companies are developing agricultural robotics as a way of overcoming labour availability issues, reducing the cost of inputs such as diesel and herbicide, and enabling the use of smaller machines that compact the soil less than the large tractors we see commonly used today.

Medical robots

In the Star Wars movies, medical droids appear at critical moments. The medical droids 2-1B and FX-7 twice patched up Luke in The Empire Strikes Back. Once when he survived the Wampa attack on Hoth and then again at the end when they grafted on a robotic hand to Luke after his father sliced it off.

Similar Imperial DD-13 medical droids created the droid-like Darth Vader from his battered body following his light sabre duel with Obi Wan on the volcanic plant Mustafar in Revenge of the Sith.

An EW-3 midwife droid even helped Padmé give birth to the twins Luke and Leia just prior to her tragic death.

Here on Earth, Google has been talking about its plans for new medical robots. It’s teaming up with medical device companies to develop new robotic assistants for minimally invasive surgery.

Medical robotic assistants have already become a common sight in well-equipped modern hospitals and are being used to help surgeons during urology procedures and more recently for knee replacements. New research is also showing how novel tentacle-like robot arms may be used to get to difficult to reach places.

The hope is that medical robotics will enable shorter training times for surgeons, lengthen a surgeon’s career and improve outcomes for patients. All these benefits could drive the cost of these procedures down, giving access to more people around the world.

Killer robots

Unsurprisingly, there are many droids in the Star Wars universe dedicated to killing. In Episodes I-III, the Trade Federation used droid starfighters. These were spaceships that were droids themselves and the droid command ships housed thousands of them.

The Trade Federation were also fans of deploying thousands of humanoid shaped B1 Battle Droids. Although they were relatively well equipped, they seemed stupid and were even worse shots than Stormtroopers. The far more capable Destroyer Droids had deflector shields and rapid fire laser cannons.

Killer robots and their development is a hot topic right now on Earth. A campaign has been started with the aim of developing arms controls and some killer robots have already been deployed.

In the Middle East, drones are routinely used to deliver missiles. These are human controlled and are not autonomous but they are changing the face of conflict.

In the DMZ between the Koreas you will find fully autonomous robots equipped with heavy duty, long-range machine guns. If they spot movement in the DMZ they are capable of firing. There is no need for a human in the command chain. They are real Destroyer Droids.

Are these the utility droids you’re looking for?
Flickr/donsolo, CC BY-NC-SA

What is missing?

Even though we can see many examples of how the droids of Star Wars may have inspired the design of the robots of today, there is one major missing piece of technology that means our robots are nothing like a Star Wars droid. And that is the almost complete lack of reliable and capable artificial intelligence in our robots.

Nearly all of the human created robots that I have mentioned rely entirely on a human expert to either control them remotely or program them to do a small range of very specific tasks. The robots that we have today are not very autonomous.

Most of them cannot see, and even if they could, engineers have yet to develop artificial intelligence to the point where a robot by itself could solve a meaningful problem it may encounter in the world.

Really smart robots are coming and many people are working hard to tackle the challenges but we are not likely to see general-purpose droids in the near future. We have a long time to go, and are far, far away from welcoming cute robot companions such as R2-D2 and BB-8 into our homes and workplaces. Until then, let’s just all enjoy Star Wars.

The Conversation

Jonathan Roberts, Professor in Robotics, Queensland University of Technology

This article was originally published on The Conversation. Read the original article.

Sea level rise is real – which is why we need to retreat from unrealistic advice

In the aftermath of 2012’s deadly Hurricane Sandy, New York launched a US$20 billion plan to defend the city against future storms as well as rising sea levels. David Shankbone/Flickr, CC BY

Mark Gibbs, Queensland University of Technology

Coastal communities around the world are being increasingly exposed to the hazards of rising sea levels, with global sea levels found to be rising faster over the past two decades than for the bulk of the 20th century.

But managing the impacts of rising seas for some communities is being made more difficult by the actions of governments, homeowners – and even some well-intentioned climate adaptation practitioners.

Coastal adaptation policies usually carry political risk. One of the main risks is when communities end up divided between those wanting a response to the growing risks of coastal flooding, and those more concerned about how their own property values or insurance premiums might be hit in the short-term by such action. For some, the biggest threat is seen to be from sea level rise adaptation policies rather than sea level rise itself.

Some organisations and governments have side-stepped the political risk by commissioning or preparing adaptation plans – but then not implementing them.

A colleague of mine describes this as the “plan and forget” approach to coastal adaptation. It’s all too common, not only here in Australia but internationally. And it can be worse than completely ignoring the risk, because local communities are given the impression that the risk is being managed, when in fact it is not.

[iframe width=”640″ height=”360″ src=”https://www.youtube.com/embed/wZXbpBEfojU?feature=player_embedded” frameborder=”0″ allowfullscreen>

The Australian Broadcasting Corporation’s Catalyst program examines past and future sea level rise.

‘The road to hell is paved with good intentions’

Coastal adaptation researchers and practitioners (and I’m one of them) must reconsider some of the common recommendations typically contained in coastal adaptation studies.

In my experience, well-intentioned but poorly considered recommendations – such as advocating for highly urbanised city centres to be relocated inland – prevent many adaptation studies being implemented.

Relocating buildings and other built infrastructure further away from the coast to reduce or eliminate the risk of flooding might sound like a sensible, long-term option, and indeed it is in some cases.

But too often, the advice given to “retreat” or relocate established, highly built-up city blocks makes little economic or practical sense. Such advice can be inconsistent with well-established engineering disaster risk reduction frameworks such as Engineers Australia’s Climate Change Adaptation Guidelines in Coastal Management and Planning.

Much to the chagrin of many in the coastal adaptation science community, cities and owners of major coastal facilities around the world are voting with their feet – largely rejecting coastal retreat recommendations in favour of coastal protection.

Major cities choosing defence, not retreat

New York is perhaps the best example of governments and individuals alike choosing protection rather than retreat.

In October 2012, Hurricane Sandy left behind a trail of destruction of more than US$71 billion in the United States. In New York alone, 43 people were killed.

In June 2013, then Mayor Mike Bloomberg said rising temperatures and sea levels were only making it harder to defend New York, warning:

We expect that by mid-century up to one-quarter of all of New York City’s land area, where 800,000 residents live today, will be in the floodplain. If we do nothing, more than 40 miles of our waterfront could see flooding on a regular basis, just during normal high tides.

Yet even after acknowledging that threat, New York’s response wasn’t to retreat. Instead, the mayor launched a US$20 billion plan to protect the city with more flood walls, stronger infrastructure and renovated buildings. As that “Stronger, More Resilient New York” plan declared:

We can fight for and rebuild what was lost, fortify the shoreline,
and develop waterfront areas for the benefit of all New Yorkers. The city cannot, and will not, retreat.

Similarly, none of the winners of Rebuild By Design – an international competition to make New York and surrounding regions more resilient to coastal inundation – focused on retreat strategies. In fact, some involve intensifying urban areas that were under water during Hurricane Sandy.

In the worst hit areas, even when given the choice of a state buy-out scheme relatively few New Yorkers chose to leave.

[iframe width=”640″ height=”360″ src=”https://www.youtube.com/embed/1pW5MZFU0E8?feature=player_embedded” frameborder=”0″ allowfullscreen>

PBS Newshour looks at how New York and other world cities can better protect against rising seas and storm surges.

Although not directly related to climate change, the Japanese response to the devastating 2011 tsunami is another telling example.

There, some residents did choose to relocate to higher ground. However, the government did not relocate major facilities inland, including the Fukushima nuclear facility. Instead, Japan will spend US$6.8 billion to form a 400-kilometre-long chain of sea walls, towering up to four storeys high in some places.

[iframe width=”640″ height=”360″ src=”https://www.youtube.com/embed/AXhjXkd5O7U?feature=player_embedded” frameborder=”0″ allowfullscreen>

In Melbourne, Australia, four local councils from the Association of Bayside Municipalities worked on the science-based Port Phillip Bay Coastal Adaptation Pathways Project to systematically identify the most effective adaptation responses. That project highlighted the effectiveness of accommodating and reducing flooding through established engineering approaches.

For example, the project concluded that while the popular Southbank waterfront in the City of Melbourne is likely to see even more common and extreme flooding in the coming decades, “retreat is not necessary”.

The Yarra River flows through the heart of Melbourne, in Australia, with Southbank on the left.
R Reeve/Flicker, CC BY-ND

More practical advice is crucial for greater action

Coastal adaptation studies and plans need to be based on practical, defensible and implementable recommendations.

That means climate adaptation practitioners need to refrain from recommending that major urbanised coastal centres be relocated further inland in coming decades, unless that really is the only viable option.

Instead, I think we can achieve more by concentrating more on how lower- and medium-density coastal communities can adapt to higher sea levels. This is a more challenging problem, as economic analyses can produce very different recommendations depending on which so-called “externalities” are included or left out in the analysis.

On the same note, adaptation studies that make recommendations without considering the impacts to present-day home-owners, or how adaptation plans are financed, can also be unhelpful.

Florida, USA, photographed from space – one of many highly urbanised coastal areas around the world needing to adapt to rising seas.
NASA

Good adaptation strategies need to acknowledge the real political risks involved with any change involving people and property. Along with making recommendations, they also need to lay out an implementation plan showing how individual and community concerns will be taken into account.

So far the climate models have done a good job in estimating the likely future sea levels. The same cannot be said for our adaptation responses.

But if you’re looking for examples of how we can be better prepared for growing sea level risks, initiatives such as the Port Phillip Bay Coastal Adaptation Pathways Project and the Queensland Climate Adaptation Strategy (currently under development) seem to be heading in the right direction.

The Conversation

Mark Gibbs, Director: Knowledge to innovation, Queensland University of Technology

This article was originally published on The Conversation. Read the original article.