Content That Leads: How AI Can Help You Stay at the Forefront of Your Industry
Getting article ideas from the leading lights in your industry
When people think of AI, creativity is not the first thing that comes to mind. However, AI has proven itself to be an incredibly effective brainstormer. A study by professors at the Wharton School of the University of Pennsylvania asked both MBA students and ChatGPT-4 to generate new product ideas costing less than $50 that would appeal to college-aged consumers. Among the top 40 most desirable products identified in the experiment, 35 were generated by the AI, while only 5 came from human students.
But when it comes to identifying good content ideas for niche audiences, AI is likely to fail. AI models lack unique insight into your audience and, absent additional context, will suggest generic topics that will fall flat. To remedy this, I created a solution that pulls down content from thought leaders in a given space, feeds that data to ChatGPT, and produces a final report with niche-specific, cutting-edge content ideas. Anyone not interested in reading the full article can find a summary here.
Step 1: Identify where the thought leaders in your space hangout
Given my interest in product, go-to-market (GTM) and AI, the thought leaders in my space are almost all on LinkedIn and X.com. Many are also on YouTube, Substack, and Beehiiv, but LinkedIn and X.com are still primary.
Step 2: Find the accounts of the thought leaders
Ideally, you already have a few accounts that you actively follow and get inspiration from, which can serve as a useful source of insights. If you are unsure of who the thought leaders in the space are or want to flesh out your existing list of accounts, you can use ChatGPT for advice.
However, I encourage you to be as precise as possible about your niche and share some sample accounts. For example, “I am interested in go-to-market for B2B startups, and want you to identify LinkedIn accounts similar to Kyle Poyar, Maja Voje, and Elena Verna.” Without this level of specificity, ChatGPT will likely recommend individuals who are brilliant self-promoters but may be lacking in substance (e.g. Grant Cardone or Andy Elliot).
The accounts I selected for content inspiration were the following:
X.com Accounts
johnrushx
wes_bush
blakeandersonw
levelsio
deedydas
mckaywrigley
LinkedIn Accounts
https://www.linkedin.com/in/elenaverna/
https://www.linkedin.com/in/majavoje/
https://www.linkedin.com/in/kyle-poyar/
https://www.linkedin.com/in/yashtekriwal/
https://www.linkedin.com/in/gisenberg/
https://www.linkedin.com/in/arjunmoorthy/
https://www.linkedin.com/in/retentionadam/
https://www.linkedin.com/in/lennyrachitsky
Step 3: Identify the right tool for scraping
Given that I had settled on LinkedIn and X.com as my platforms of choice, I wanted to find a scraping solution that would enable me to pull down my data as cost-effectively as possible. Ultimately, I found this Apify scraper for X.com and this Phantombuster scraper for LinkedIn.
I went with Phantombuster for LinkedIn since I already have an account with Phantombuster and get a certain number of scraping hours per month, so there was no additional cost associated with scraping this data. I went with Apify because their X.com scraper is very cost-effective, enabling you to scrape 1,000 tweets for $.40. Phantombuster also has X.com scrapers, but I have had issues with them in the past.
Step 4: Scrape the content
Once I knew which platforms to focus on and which accounts to scrape, scraping the content was straightforward. The scraping process and output can be seen in the video below.
Step 5: Batch the data (if needed) and feed it into ChatGPT
In my last article, I talked in detail about ChatGPT’s context window and how it will start forgetting key parts of the thread if the number of tokens is exceeded. The maximum amount of content that ChatGPT can comfortably ingest is about 190 pages of content (sometimes, it can be a bit more depending on the number of words per page), so if you have more than that, you may need to break it up and feed each piece of content into ChatGPT sequentially (a process known as batching).
In aggregate, the LinkedIn and X.com accounts I selected had over 1,000 pages of content to scrape, so batching was a necessity. I broke the Twitter posts I scraped into two files, here and here, and the LinkedIn posts into three separate files.
Step 6: Combine the output into a final report
When I fed the data into ChatGPT, I used two chats: “Thought Leader Content Ideas - Twitter” and “Thought Leader Content Ideas - LinkedIn.” Once it produced some answers, I needed to combine the output into a final report. I did that and ultimately produced this as my final output.
While this list won’t tell you exactly what to say for each topic (it is still important you write a well-researched article with a fresh perspective), it will certainly tell you the most cutting-edge ideas and points of discussion in your niche. Amazingly, by feeding content from just a few accounts as inputs to ChatGPT, I produced 23 pages of article ideas.
Conclusion
AI has proven to be an invaluable tool for brainstorming and generating ideas. Still, it requires the right approach to truly unlock its potential. By leveraging scraping tools and feeding data from thought leaders directly into ChatGPT, you can access a goldmine of niche-specific, cutting-edge content ideas that would take hours or even days to gather manually. While AI can’t replace the need for a unique voice or fresh perspective in your writing, it can streamline the ideation process and help you stay ahead of the curve.
If you liked this content, please click the <3 button on Substack so I know which content to double down on.
TLDR Summary
This article outlines a solution for generating niche-specific, cutting-edge content ideas using AI. By scraping content from thought leaders in a particular space (like LinkedIn and X.com), feeding that data into ChatGPT, and combining the output, you can efficiently generate valuable content ideas that are tailored to your audience. This method leverages AI’s strengths in brainstorming while ensuring the final product remains unique and insightful.
Key Steps and Insights
Identify Where Thought Leaders Hang Out
Focused on LinkedIn and X.com as the primary platforms for scraping.
AI can help identify additional thought leaders if needed.
Find the Accounts of Thought Leaders
Use ChatGPT to refine searches and identify the right niche-specific accounts.
Be specific when asking AI to help identify accounts to avoid generic suggestions.
Select the Right Tool for Scraping
Used Apify for X.com scraping and Phantombuster for LinkedIn due to cost-effectiveness and familiarity with the tools.
Apify is efficient for scraping large volumes of content from X.com.
Scrape the Content
Scraping content from selected accounts was straightforward.
The process can be automated for efficient data extraction.
Batch the Data and Feed It into ChatGPT
Used batching to handle large volumes of data due to ChatGPT's token limitations.
Broke the data into smaller parts to ensure ChatGPT could process it effectively.
Combine the Output into a Final Report
Combined insights from LinkedIn and X.com into a comprehensive report.
The result was a collection of 23 pages of content ideas that could be used as a foundation for articles.
Conclusion
AI can significantly enhance the brainstorming process by generating relevant, niche-specific content ideas quickly. By combining scraping tools with ChatGPT, you can automate the discovery of cutting-edge topics while maintaining a unique voice in your writing. This method saves time, offers fresh perspectives, and keeps you ahead of trends in your field.
This is an incredibly smart and efficient approach to content ideation! Leveraging AI to analyze thought leaders’ insights and generate niche-specific ideas is a game-changer for staying ahead of industry trends!