Exploring the Bing conversational search experience

John John Clyman

Recently, we wrote about the transformational potential of generative AI tools like ChatGPT and what this might mean for creators.

Today, we’ll share some hands-on examples of Bing’s new conversational search capabilities and what this foreshadows for SEO. Right now, the experience is clunky and limited, but it’s quickly evolving and certainly poses a future threat if Bing’s share of search queries grows or Google’s search experience follows suit.

As your advocate in a complex and fast-paced digital world, we’re listening to creators’ concerns, staying on the cutting edge of these search developments, and working on innovative ways to protect your interests and help you thrive in the search experience of tomorrow – whatever that may be.

While Microsoft aims to widen its reach dramatically in the future, at the moment, Bing accounts for just 3.5% of search sessions reaching our creators’ sites as a whole, and the chat experience is only available in limited circumstances with frequent iterations.

But let’s be realistic. What Microsoft does probably isn’t going to affect most creators too massively, as long as Bing and other Microsoft search capabilities remain niche players.

The real question is: what will Google do?

Given its dominant search share and corresponding focus in everyone’s SEO efforts, the bigger issue is what Google might do with conversational search.

News reports indicate (note: subscription required) that Google is concerned about whether something like Bard can really be an effective search experience and how it would impact the gigantic business it supports.

In our opinion, it’s hard for Google to make a truly radical move in short order. Google generates tens of billions of dollars a year in revenue from sites that get clicks from searches – both from the ads it places on those sites and from its role as a preeminent DSP (demand-side platform) placing third-party ads on those sites. If Google search stops sending substantial volumes of traffic to unaffiliated websites, Google itself could be badly hurt.

But will that be enough to prevent Google from deploying an “answer engine” that makes most clickthroughs to third-party sites unnecessary? That’s an understandable fear, and it’s certainly something we are thinking about.

While we can’t be sure what the future holds, we expect Google to move cautiously. As the underdog, Microsoft has little to lose and can afford to be bolder and potentially faster. Still, what they’ve previewed of their ChatGPT integration is far from doing away with organic search results at this point (although that too could change.)

Quick-take for creators

Conversational search is in its infancy. We remain optimistic that high-quality independent sites can continue to thrive while keeping a watchful eye on potential risks and playing an active role in mitigating them as much as possible. We’ll continue to update you on developments and advocate for your rights in this area.

What you can do

Lean into your humanity and build trust and rapport with your audiences.

For now, we encourage sticking with the same kind of ethical SEO techniques that have long proven effective.

  • Identify the type of content that your visitors want most.
  • Write about it conveying your human expertise, experience, authoritativeness, and trustworthiness (Google’s EEAT criteria).
  • Use tools like Topic to help you brainstorm potential approaches and see all the angles.
  • And monitor the impact of your actions to learn what works for your site and continue building on your successes.

Consider hedges like shifting some effort into rich media like video, which monetizes highly and which conversational AIs might link to but can’t easily replace.

What we’re doing

We’re not just sitting here waiting to see what happens. We continue to work closely with these capabilities, both to keep evolving our products and to stay abreast of how they might enable disruption – much like we have with the projected elimination of third-party cookies.

We will continue reinforcing the message that independent content creators and the open web are invaluable. And we’ll be announcing more soon about some of our efforts to make sure creators have a say in how their content is used by AI.

With that framework in mind, let’s take a deeper look at Bing’s conversational search experience as it stands today.

The Bing conversational search experience on desktop

Bing’s new interface invites visitors to “Ask me anything”, with some sample queries:

Screenshot of Bing search experience on desktop

If you ignore the invitation to be chatty and type a typical search query to get a baseline comparison, you’ll see search results that look pretty traditional.

Screenshot of Bing search results on desktop

To actually engage with the search bot in Bing on the desktop, you can click on the “Try the new Bing today” link on the upper right of the search results page:

Alternatively, you can click on the Chat link in the bar above search results, by images, videos, news, and such.

When you click into chat, you get a different experience that feels grafted onto the basic search capability rather than seamlessly integrated. This could well change in the future, but right now it essentially means users need to make an explicit choice to use the chat interface.

Getting talkative

Even if you use a more conversational query from the start, you’ll typically still get traditional search results, albeit sometimes with a widget on the right that has a conversational summary:

Screenshot of Bing chat search results

A closer look inside that box:

Screenshot of Bing chat search results box

In a different presentation, Bing will show an inline summary (which doesn’t seem to be generated by ChatGPT) in search results, with an invitation to begin a free-text chat.

Either way, as far as we’ve seen at this point, Bing doesn’t appear to be pre-empting organic search outright. Instead, it always provides some organic search results until we explicitly switch it into chat mode.

Let’s chat

OK, so you’ve told Bing you really do want to engage in a conversation. What’s the experience like now?

When you start the chat mode and enter a prompt, Bing first converts it to a query and informs you that it’s searching. Then it begins to synthesize a result.

The thing you can’t miss here is that the process is slow. Really slow. It typically takes Bing about 10 seconds to complete the search and begin generating answers, then another 15–30+ seconds to actually write out the response for you. If you’re used to sub-second search responses, this can feel like an eternity!

Maybe this will get faster in the future, but at this point, it’s a pretty striking downside to using this interface.

Access to GPT models programmatically via API is much faster in our experience, so it remains to be seen whether performance improves or is a fundamental limitation of integrating search with these large language models.

If you ask a good question, you might get a decent initial answer, typically with links to relevant websites (though the quality of these links is often mixed at best). But often the answer is relatively general, or only addresses part of your inquiry.

Screenshot of Bing chat interface

Many times you’ll want or need to further refine the results, which is where the chat interface ought to shine.

Bing gave us some pointers here, but it still hasn’t answered the original question, “What should I pack for a trip to Santorini?”

Let’s ask point blank:

At this point, we’ve been able to coerce Bing chat into giving us an answer that might be thorough enough to substitute for visiting a website with actual expertise. Even so, it’s not obvious that this experience is an improvement over traditional search.

Getting the results we wanted required multiple exchanges, waiting patiently each time, and putting up with digressions.

And of course, the chat results lack the context and commentary that helps build the confidence and trust you get from visiting an actual creator’s site.

This is just one sample conversation, but it illustrates how Bing and ChatGPT aren’t yet at a point where every typical search can be seamlessly replaced by an “answer bot,” even if you wanted that.

Moving to the Bing conversational search experience on mobile

More than half of traffic to most of our creators’ sites comes from mobile devices, so let’s also look at the Bing chat experience there.

With the Bing mobile app, the emphasis on the chat experience tends to be more prominent. On the opening screen:

When you click the Bing “b” in the quote bubble:

The microphone icon is prominent so you can use text-to-speech to capture a more conversational query if you’d prefer to bypass your keyboard.

Once you’re in chat mode in the mobile app, the experience is similar to desktop, with one big exception — on mobile, the results often include related videos interspersed with conversational results. This does make the results feel more organic and engaging.

Interestingly, in our testing so far, the only videos we’ve seen appear are from YouTube, which isn’t even the case if you search for videos from Google itself.

To be sure, there aren’t any easy answers to how, exactly, conversational search may shift SEO and traffic in the future.

But in more than a decade of working with many of the web’s leading independent creators, we’ve weathered plenty of storms together. Search algorithms and zero-click capabilities have changed. Technology has closed some doors and opened others.

That will continue to happen, and we’ll continue to stay on top of it, inform you about risks and opportunities, and work to keep you empowered to continue growing successful sites.