For those who lack a technical background it is difficult to wade through the various technologies available for a website.  There a numerous considerations including where to buy a domain, where to host a site, where to set up email, and what toolkits to build your website in.

Domain registrars
Let's talk about purchasing a domain first.  Your domain name is your online identity and will uniquely identify your website.  You may already own your domain name and that's great but let's assume you haven't gotten that far.  The first thing to understand is how domains are purchased and managed.  A domain name registrar is a service that allows you to register your desired website domain name so no one else can own it or use it.  There are literally hundreds of domain registrars but if you've ever watched the Super Bowl the one you're probably most familiar with is GoDaddy.  Registrars can differ in a variety of ways including price and customer support.  These are things to consider when choosing a registrar.  I'm not going to provide a recommended list but instead will say I personally use GoDaddy.  They have great customer support, their prices are reasonable and because they are a leader in the industry there are endless instructions online about how to integrate your hosting service with GoDaddy.  If you're looking for simplicity this is the way to go.

Next is hosting a site.  You own a domain name but that doesn't mean you have a site.  You first have to build a site and that site has to be hosted somewhere meaning someone somewhere is providing you with the hardware (a server) to store your site.  The biggest sales pitch many hosting sites give is the reliability of their servers and having a 99% uptime.  They are basically telling you your website will never be down.  This is obviously important because if people can't open your site they definitely aren't going to become a customer.  Price should also be considered.  Price can be directly related to the types of services and hardware provided by a hosting service.  Many times you get what you pay for.  Quality hardware and customer support cost money.  A cheap hosting service can be a signal they aren't providing this.  Different hosting services often cater to a specific audience.  Some are tailored for very large enterprises while others are designed for small personal blogs.  These differences are usually due to the technical requirements a site has.  If your site is going to be heavy with e-commerce, images, and videos you need to make sure your hosting service provides adequate memory, processing power, and hard disk space to accommodate these things.  A website that is slow to load or crashes because your hosting service can't handle your images and videos reduces the user experience and will cause many people to leave your site before they have a chance to look at anything.  Remember, we live in a fast paced world and expect instant gratification. 
I've described the basics of hosting services but one thing to note is if you use a website builder such as Squarespace or Wordpress, hosting will be included.  For those with limited technical skills this is great because you can design and host your site all in one place without the anxiety of choosing a hosting service.  Customer support is also consolidated so whether you have questions about design changes or problems with site performance you can find help all in one spot.

Email is relatively simple.  Oftentimes your domain registrar will offer email hosting.  GoDaddy does and it works just fine.  Google also has a service called Google Apps where you can set up a custom email domain.  Your web hosting service may also provide email options.  Overall this is the least difficult part of the process.

Website builders
Now it's time to choose what to build our website in.  You can hire a web designer to create a fully customized solution for you.  This often comes at a price though.  A custom website could cost you between $10,000 and $100,000 depending on the features you request.  For most small businesses this can be a huge roadblock to getting a website up and running.  Another downside to a custom design is the inability to hand off the site to a business owner.  Often you can be forever reliant on the designer when issues with the site arise or changes need to be made to features or content.  This can end up costing a lot of money because a designer is likely to charge you for their services any time a change request is made.
So rather than discussing custom solutions I am going to talk about website builders.  All the information I am providing is aimed at small businesses that don't have deep technical knowledge.  You need a simple solution that fits into your budget and allows you to easily manage the site yourself.  There are quite a few options available such as Squarespace, Wordpress, and Shopify.  All of these tools are designed for people with a limited technical background.  Some desirable features to look for are drag and drop capability, an easy intuitive interface, customization, professional looking templates, SEO friendliness, mobile compatibility, and easy integration with social media, widgets, and mail services such as Mailchimp.
I specialize in custom Squarespace design because I believe it is one of the best options available for a small business.  It has beautiful modern, clean templates and meets all of the requirements I listed above.  It also implements a responsive design which means your site will change based on the device it is being viewed on.  This is wonderful because in the past you needed to build separate versions of your site if you wanted it to look right on different devices. Probably my biggest appeal for a tool like Squarespace is that unlike many custom website builds it can be handed off and easily managed with very little technical expertise.