Even when you explain that it's a country, the question is the same: "Where?"

It's in Central Asia, you say. Russia to the north. Iran to the south. The Caspian Sea to the east, Armenia and Georgia to the west. That's when the penny begins to drop and you can prepare to be faced with the inevitable follow-up question about our travels: "Why?"

Now this is the interesting one. Why would you want to go to Azerbaijan, a former Soviet state, an oil-rich and tourist-poor country that straddles Europe, Asia and the Middle East, a place of sometimes troubled history, but with the conviction of a bright future, a place of development, a place of natural beauty, a place no one really knows anything about?

Even I am asking myself that question as I make my way towards Baku, the Azerbaijan capital, on a flight from Qatar's capital of Doha. There's a Westerner sitting next to me, a guy called Paul, who it turns out has a similar story to many of the Westerners going to Baku: he works on the rigs. There's oil and natural gas in Azerbaijan, plenty of it, and it attracts those looking to make a buck. Like Azerbaijan itself, this whole trip is something of a mystery to me. I've been invited to a place I've barely heard of, let alone could pinpoint on a map, by an entity who remains shadowy at best. At first I think it's the Azerbaijani government; then I'm told I'm being hosted by a hotel chain; finally I figure out, through a game of Chinese whispers with other journalists, that the invitation came from one particular hotel board member who has a passion for spreading the word on Azerbaijan.

This whole trip will prove to be a discombobulating experience, an exercise in altered perceptions and genuine confusion. Here's the thing: Azerbaijan is a little bit weird. It's not really old, it's not really new, it's not really Islamic, it's not really secular, it's not European, it's not Asian, it's not beautiful and it's not ugly. Although sometimes it's all of those things. And sometimes it's none.

What is this place, you wonder, as you stroll around town? There's a shuttle bus that takes visitors from the Kempinski Hotel, where I'm staying, into the centre of Baku. However, for my first journey I elect to walk. What better way to get to know a new city than striding its avenues and breathing its air?

Only trouble is, the air in Baku is polluted – heavily. There's also not much to see between the Kempinski and the middle of town save for endless rows of dreary post-communist apartment blocks and a huge graveyard. But hey, this is different, this is interesting. People in cars are waving at me. It feels friendly.

And then I arrive in the centre of the city, down by the shore of the Caspian Sea, and the appeal of Baku reveals itself. It's Rome, it's Paris, it's London. Back in the early 20th century, Azerbaijan went through its first oil boom, and architects from all over Western Europe were commissioned to design buildings for the expanding city. The result is a charming mix of architectural styles, genuinely beautiful buildings that line the city's traffic-choked streets.

Towering above them all is perhaps Baku's most impressive sight, the Flame Towers. Inspired by Azerbaijan's nickname, the "Land of Fires", these three flame-shaped towers dominate the skyline, and have become something of a national symbol since they appeared in 2012. By night, huge LED displays give the appearance of flames licking the sides of the buildings.

Azerbaijan, you quickly realise, has money. Down the bottom of one of the Flame Towers there's a Lamborghini shop. Wander around near the lake and you'll find names like Gucci, Tom Ford, Bulgari and Tiffany slapped across the storefronts.

The natural resource boom may not have trickled down to all parts of society, but it's certainly propping up the big end of town, so much so that Azerbaijan seems determined to bill itself as a sort of "new Dubai". To do that you need impressive buildings, which Baku is on its way to achieving. You also need to host important events, which brings me to another reason we've all been invited here.

There's sport happening in Azerbaijan. While Azeris themselves are known for their prowess in combat sports like wrestling and boxing, and for their victory in the 2011 Eurovision Song Contest (if you can count that as sport), there are plenty of other contests of equally questionable legitimacy being hosted in the Central Asian country.

This year Baku staged the inaugural European Games, a sort of faux Olympics that features sports such as three-on-three basketball and beach football, along with the more familiar wrestling and boxing. There are also regular GT touring car meets here, and the city was poised to host a Formula 1 Grand Prix this year, though that's been postponed until 2016.

If you build it, they will come – right? The tourists that is. That's the impetus for many of these buildings, shops and events. They will also come if you make it easier for them to come, which is why Azerbaijan has relaxed its tight entry requirements, allowing tourists to apply online for an e-visa rather than post passports to an embassy. More than two million people are now venturing to the country each year.

Some, obviously, will come for the sporting events. Others will come for the concerts that are held regularly in the Baku Crystal Hall, which was built to host the 2012 Eurovision Song Contest – they're probably still finding sequins in odd places in there. Others will come to experience a rich culture that finds its expressions in food, and in the intricately woven carpets that are sold by dealers throughout Baku. More still will come to wander through Baku's charming Old Town, with its 1000-year-old Maiden Tower and its ancient caravanserais.

There are some who will come here for the natural beauty alone. While Baku might not be much to look at in some places, Azeris boast that their country is home to nine of the world's 11 climate zones, from semi-desert in the central lowlands to rainforests in the northern hills and Alpine tundra in the Greater Caucasus.

You can scuba dive in the Caspian Sea, sun yourself on a beach, or ski in resorts in the nation's north.

Azerbaijani food, with its mix of influences from Iran, Turkey and Georgia, is unfairly anonymous on the world scene, but here you get to try it in all its comforting glory. You can also sample the caviar. Famed beluga caviar from the Caspian Sea is treasured here and regulated even more zealously than oil – however, walk into any market and you'll hear soft calls of "caviar, caviar" from black market sellers seeking to offload their goods.

It's expensive, it's cheap. It's legal, it's dodgy. Like many things in Azerbaijan, it's confusing.

Meanwhile, my strange tour of this land continues. Our group of journalists is being ferried by bus to the European Games gymnastics venue to watch Azerbaijan's bright young things practise. We're being whipped through the National Museum of History at the speed of time travel itself.

And finally we come to the country's undoubted architectural highlight, the Heydar Aliyev Cultural Centre. Designed by British-Iraqi Zaha Hadid, this is one truly stunning building, with wave after white-tiled wave of rooftop flowing over a hill in the north-east of the city. This is what modern Azerbaijan is all about: something a little daring, something that will stick in the mind's eye.

The centre certainly does that. It may not answer the question of "Where?" when it comes to Azerbaijan. And it doesn't provide many clues as to what I'm doing here. But for prospective visitors to the country, it's certainly one good reason why.