Technical requirements

Overview¶

The Apteco Marketing Suite is a range of marketing data analysis and automation applications. It supports easy analysis, visualisation, predictive modelling, and campaign management with massive customer and prospect databases. The system has a multi-tier, web service architecture using a variety of client application technologies. The FastStats.NET Server powers the data processing and provides specialist marketing data analysis query capabilities. All data resides securely on the host system and is available only to authenticated users.

FastStats.NET server¶

The FastStats.NET Server is a highly scalable, multi-threaded, multi-server solution. It runs on Windows servers and provides an XML Web Service and API. The Server supports a wide range of system configurations, from a single laptop to a load-balanced, multi-server configuration. This setup provides no single point of failure.

The system administrator determines the connection scope of the client applications. These applications can run on a departmental or corporate Intranet, or connect to the Internet. This document provides a summary of the requirements of a server intended to run the FastStats.NET WebService on either the Internet or an Intranet.

Client applications¶

The FastStats range includes many client applications:

• Apteco FastStats: powerful analytics, visualisation, and predictive modelling
• Apteco PeopleStage: event driven, multi-channel, visual campaign management tool
• Apteco Orbit: A platform-independent, browser-based application that works on tablets, phones and desktop computers
• API: Application Programming Interface
• Excelsior: Excel plug-in for dashboard type reporting

Apteco FastStats¶

Apteco FastStats is a rich and powerful Windows.NET application supporting a highly intuitive, flexible, and powerful drag-and-drop interface. The application can be locally installed or deployed online using a small downloadable launcher application that automates version control from the web server.

The user interface includes a powerful selection mechanism with full Boolean logic across multiple tables, sampling, Top/Bottom N, and RFV criteria. The powerful analysis tools include multi-dimensional cubes, nested trees, and five-way Venn diagrams. Predictive modelling techniques include profiling, clustering, and decision trees. You can export selected subsets of records in a wide variety of formats with coded or described data values. Apteco FastStats is Apteco's premier product, providing the broadest range of functions.

Apteco PeopleStage¶

Apteco PeopleStage is an event driven multi-channel campaign management tool which provides a shared drawing space for multiple users to define marketing process flowcharts. PeopleStage enables highly personalised communications, running multiple tests concurrently in multiple campaigns across multiple online and offline channels. It also applies marketing constraints such as budget, volume, and channel. PeopleStage uses the segments, models, scores, response rules etc. defined in Apteco FastStats and produces marketing reports that you can distribute by email or save back into the analysis environment. The application also requires Excel 2007 or later, or Office 365.

Apteco Orbit¶

Apteco Orbit is a browser-based, device-independent platform that works on desktops, laptops, tablets, and phones. No client installation is necessary. Users access Orbit through a modern web browser. For server-side installation requirements, see ASP.NET Core 8 requirement.

API¶

The FastStats API enables third party developers to use FastStats results. The API supports XML, HTML, and JSON results. You can use it to retrieve up-to-date results for FastStats objects created in Apteco FastStats or Orbit. You can restrict the API to licensed users or make it available for public access.

FastStats Excelsior¶

The FastStats Excelsior plug-in embeds FastStats results in Microsoft Excel spreadsheets. You can calculate, format, and present these results using standard Excel methods such as charts, grids, and traffic lights, typically for dashboards. Excelsior supports parameters used in FastStats objects, allowing users to run variations of reports. Users may have permission to create reports, or only to refresh and publish them as plain Excel documents.

FastStats.NET Web Service architecture¶

The FastStats.NET WebService coordinates and multi-threads work to the FastStats Servers. It also receives results from the FastStats Servers and delivers the result messages back to the client applications.

The FastStats Server is a Component Object Model (COM) object deployed as a 64-bit Windows DLL. This specialist database server provides marketing data selection and analysis features. FastStats uses a compressed and indexed column oriented representation of the data stored in a proprietary format optimised for the querying work.

Scalability¶

The FastStats system scales linearly in both user count and data volume.

Users¶

The FastStats.NET web service supports single or dual web servers (for resilience) and single or multiple FastStats servers (for resilience and capacity). The system is scalable in line with the server capacity provided by multiple machines. Multi-server installations also gain maintenance flexibility, as you can close individual machines while the rest of the system stays operational.

The number of supportable users depends on the database size, and on the type and intensity of user activity. It's practical to support hundreds of users on a single server if the work undertaken isn't intensive. Larger production implementations already support thousands of users from a three-tier, fault-resilient configuration.

Data volume¶

The FastStats system is linearly scalable for data: both the data storage requirement and the query times increase linearly as the data volumes increase. It's common practice to build FastStats systems of tens or hundreds of millions of rows containing hundreds or thousands of attributes. Apteco's biggest systems have over 2 billion rows, with over 1 billion rows in a single table.

Metrics¶

The current largest system metrics are:

• Largest system has approximately 2.3 billion (2,300 million) rows
• Largest single table input data set has over 1 billion rows
• Largest Meta data driven attribute process has over 2,000 variables
• Tables tested at over 2 billion rows
• The widest input encountered had over 6,000 variables (now re-engineered!)
• The most frequent refresh is nightly
• The most FastStats databases on a pair of fault resilient servers is approximately 150
• The largest user base is approx. 70 worldwide from a single organization, and over 1,500 from multiple organisations run on a single set of servers

Configurations¶

The diagrams below show various different configurations of the FastStats.NET server and user interfaces.

The following diagrams show illustrations of different system configurations.

Configuration¶

To prepare for implementation, consider the following elements:

Intranet or internet¶

The FastStats.NET WebService can service users connected to either a private Intranet or to the public Internet. As the Apteco Marketing Suite is often provided on a hosted basis by marketing services providers, it's typically used in an Internet deployment. However, it works just as well for marketing analysis and automation of an enterprise's customer and prospect database, using an Intranet-connected server.

Single or multi server¶

The diagrams earlier in this section show a variety of single and multi-server solutions. Using multiple servers increases capacity and resilience, but doesn't affect the speed of any single query. Each query will still run on one thread on any particular server. With an appropriate configuration, you can remove all single points of failure and provide a fault-resilient solution. However, it's practical to operate the entire solution on a single server. The Apteco Marketing Suite implementations may start with a single server solution and grow as the data and use extends. You can agree the initial and planned growth path for an installation with Apteco.

Web server¶

The web server needs to be permanently connected to the network, either Internet or Intranet. It also needs to run Internet Information Server (IIS) on Windows Server 2016 at a minimum, though a later version is preferable. This web server needs a fully qualified domain name for use on the Internet, or a resolvable server name for use on an Intranet.

Control database¶

The FastStats.NET control database manages and stores user, group, and configuration details, and all user requests and results. This is a relatively low-volume, low-throughput database. Both the client and server processes poll it to detect work requests and deliver results. Apteco typically use Microsoft SQL Server Express Edition for initial deployments of this database, but you can replace it with any full SQL Server edition. The Apteco Marketing Suite doesn't require web-enabled editions of SQL Server to support web-connected users. You can license it on a processor or Client Access Licence basis. The database can run on the same machine as the FastStats Server, on a stand-alone machine, or on a cluster of machines for resilience. SQL Server 2016 is the minimum requirement, with a more recent version recommended.

Campaign management database¶

The Apteco Marketing Suite includes optional campaign management tools called Cascade and PeopleStage. In each case, the campaign management module stores information about marketing campaigns, promotions, communication history, etc. in a relational database. This database will typically use the same database server as the control database. The PeopleStage database requires a full edition (not Express) of SQL Server 2016 SP2 or later recommended.

Communications¶

All Apteco client communications can operate on standard HTTP or secure HTTPS connections. The communication doesn't require any non-standard port or firewall configuration.

Software delivery¶

The Apteco FastStats and PeopleStage software is typically delivered to user machines using a FastStats or PeopleStage Launcher. This small footprint application (less than 3MB) installs from the web server the first time the user runs the client application. It then downloads the executable application libraries. On each subsequent execution, the Launcher compares the previously downloaded libraries with those held on the web server and will download any that are newer. This lets system administrators roll out updates by simply installing the new libraries on the web server. Each user's software updates automatically whenever they next connect.

You access Orbit in a browser, with no installation necessary on a client machine. Apteco deliver updates to Orbit automatically by default, guaranteeing compatibility with the last four quarterly software releases.

Excelsior is available as an installer (installed locally on a user's machine) to offer a plugin to Excel.

Build server¶

FastStats operates on a proprietary snapshot of the source data, built using an "Extract Transform and Load" tool (FastStats Designer). FastStats database builds are typically scheduled to run automatically, daily or weekly, on a separate staging server. This server resides within your private network and needs to be able to connect to the source databases or files. The machine needs to be sufficiently powerful to rebuild or update the FastStats database in the prescribed time. You can estimate this workload once you've built a prototype database, as performance varies linearly with the size of the input data. A FastStats database is typically smaller than the source data. However, Apteco recommend configuring the build machine with five times the source data size. This accommodates the source data, a threefold source data sort area, the result area, and result deployment. For a more precise estimate of the space required, use the FastStats System Size Ready Reckoner spreadsheet available from Apteco.

You can deploy updated FastStats database snapshots to the production FastStats servers, either remotely through the WebService or directly. This depends on whether the staging and production areas are on a common private network. This deployment process lets you deploy new database generations without requiring users to save their work or log off.

FastStats Designer is available as a 64-bit application. To use this version, the Build Server must run Windows Server 2016 at a minimum, though a more recent version is preferable.

You can use Windows 10 for testing or in smaller desktop environments.

FastStats servers¶

There may be one or more FastStats servers depending on the capacity and resilience required. These machines will typically each use a local copy of the FastStats database, and will run processor and disk intensive tasks. Alternatively, you can store a single copy of the FastStats database on a Storage Area Network and share it between multiple FastStats servers.

In an Intranet deployment, you can host the FastStats Server (database engine) on the web service machine, or on another networked server. Typical host configurations start with the FastStats engine on the same machine as the web service. The engine then moves to a dedicated server if the load becomes sufficient. However, corporate architecture and security policies may mean a single machine implementation isn't advised. Any Internet-connected deployment needs separate web server and control database / FastStats servers. This maintains proper security boundaries: the FastStats WebService operates in a perimeter network, while the control database / FastStats Service operates in the private network. It's possible to use multiple FastStats Servers to provide capacity and resilience if the load warrants it. These FastStats Server instances use a polling architecture to share work approximately equally.

Security policy¶

The FastStats WebService can communicate using either an HTTP or HTTPS (SSL) connection. The system can authenticate and authorise users at both the operating system and application level. In each case, the authenticated user account determines the data, tools, and usage allowed. Established security policies may restrict this broad range of network functions. It's good practice to consult the web master, network administrators, and firewall administrator before commissioning a FastStats WebService on an existing web server.

Customisation¶

The Apteco FastStats application has fewer explicit customisation parameters, but automatically adopts the Windows Theme settings of the client machine. You can customise the start-up splash screen, login information panel, Help About screen, and logoff dialogue.

The Apteco PeopleStage application supports limited customisation as the colours and shapes are a prescribed part of the application design.

You can customise Orbit in a limited fashion: for example, you can add company logos and further branding.

Support applications¶

The Apteco Marketing Suite can use a number of support applications that provide a richer environment for the user. These are Microsoft Excel, Word and PowerPoint for train of thought logging and Bing Mapping for geographic mapping, drive-time selection and map generation. The user must license these applications themselves.

Email despatch¶

The Apteco Marketing Suite includes the capability to despatch emails from the server (for example in data vending installations or for password reminders etc.). This capability operates with a Simple Mail Transfer Protocol (SMTP) Mail Service and a permanent outbound network connection.

PeopleStage and FastStats integrate with external digital delivery platforms (email broadcasters) to provide bulk communication capabilities. Apteco develop these integrations and configure them through the administration suite. The integrations use a variety of connection types, including HTTP, HTTPS, Simple Object Access Protocol (SOAP), RESTful, FTP, FTP Secure (FTPS), and SFTP. These connect to the digital delivery platform API and upload lists. FastStats includes a Response Gatherer application, which retrieves batches of responses from the digital delivery platform. It processes them into a standard format in the FastStats response database.

Host capacity planning¶

There are several aspects of capacity to consider.

Bandwidth¶

The Web Server needs to be able to deliver the FastStats and PeopleStage libraries (approximately 300MB each) or FastStats Orbit application on demand. The downloads for each only occur on the first request and after software updates.

Normal use of the applications requires bandwidth to support the communication and file transfer needs of the maximum forecast number of concurrent users. Interactive use of the application doesn't normally use significant bandwidth. However, users may request downloads of data extracted by the application. The system may zip and encrypt this data, which may be tens of megabytes. A 1Mb connection is the minimum sensible, and anything higher than that will improve the performance of the system. However, most Intranets have more than enough bandwidth to support FastStats/PeopleStage, as even a 10Mbs connection would support many users.

FastStats server processor capacity¶

The faster the processor, the faster it can handle each query. The most direct influence on perceived speed of operation is the performance of the querying engine. Processor speed, cache size, and the speed of the disks holding the data files all affect this. It's best to have a well-balanced system rather than, for example, a blazing fast processor and slow disks. The FastStats Server components are extensively multi-threaded and use parallel processing for some operations (such as sorting). This means the application can benefit from a multi-core / multi-processor server.

FastStats performs relatively few but long running queries. Each query will use a maximum of a single thread for the length of the query. A query can take several minutes to run. This makes the performance profile different from that of a web server or a transactional database.

The speed of a FastStats query depends on the single-thread speed of a CPU. More threads or cores let more queries run simultaneously, and can speed up a single query if you enable distributed processing. The recommendation is to use fewer, faster cores and to choose hardware that maximises single-thread performance.

The following linked chart shows CPUs benchmarked by their single-thread performance:

https://www.cpubenchmark.net/singleThread.html

FastStats queries access a large data structure held in memory. Every query performs a sequential scan of this data structure and this data structure is far too big to fit into a processor cache. A limiting factor on performance, therefore, can be the bandwidth to uncached memory and the access time to uncached memory.

The following two linked charts show memory read speeds for Intel and AMD processors:

https://www.memorybenchmark.net/read_uncached_ddr3_intel.html

https://www.memorybenchmark.net/read_uncached_ddr3_amd.html

The following two linked charts show memory latency for Intel and AMD processors:

https://www.memorybenchmark.net/latency_ddr3_intel.html

https://www.memorybenchmark.net/latency_ddr3_amd.html

At the time of writing these charts clearly show a large difference in memory bandwidth and latency between Intel and AMD processor architectures.

While AMD processors can provide good performance, they achieve this by providing more cores and threads than equivalently priced Intel processors. This is ideal for transactional databases or for web servers that can parallelise their work effectively, but isn't the required performance characteristic for FastStats. Apteco recommend the use of Intel processors for FastStats servers.

FastStats server disk capacity¶

The servers must be capable of storing the compressed FastStats data files. Depending on configuration, these are typically between the same size as the original data and one-tenth its size. Once you've devised a configuration, you can estimate the size of the FastStats system. Use the FastStats System Size Ready Reckoner spreadsheet, available from Apteco. The FastStats data files tend to grow linearly as the number of records increases and are therefore easy to forecast. The specification of the disk devices can impact FastStats performance, as this is often constrained by the throughput achieved by the disks. It's beneficial to use a Redundant Array of Independent Disks (RAID) 5 array of disks to maximise the data throughput.

FastStats server memory capacity¶

The FastStats database engine doesn't rely on the RAM capacity of the server to deliver speed. However, more RAM lets a server running several processes run more smoothly. Using the 64-bit version of the FastStats Server components lets the server use all the RAM installed. You can configure a 64-bit Windows server with 8Gb RAM or more to enable maximum caching benefit, etc.

Starter configuration¶

For a typical Intranet single machine starter configuration, Apteco recommend:

• Core i5 or better processor
• 8 GB of RAM
• 100 GB disk storage
• Windows 2016 Server Operating System (or later) with latest service packs and updates
• 100Mbps network card
• Backup device

Full configuration¶

For a typical multi-server configuration for Internet deployment, Apteco recommend:

Web tier¶

FastStats WebService operating in IIS Web Server (one required, second instance is optional and only required for fault resilience)

• Core i5 or better
• 8GB RAM or more
• Disk Space - 110MB
• Windows Server 2016 Standard or later with latest service packs and updates
• 100Mbps network card
• Minimal backup capability (e.g. Initial backup through removable device, subsequently only need to backup log files etc.)
• If using two web servers for resilience, you'll require a load balancer (e.g. Alteon, Big Iron, etc.) and temporary shared storage, such as Network Attached Storage (NAS), for files transferred by users.
• Microsoft .NET 4.8 (minimum) with the latest version recommended.

Application tier¶

The Application Tier consists of an SQL "control" database that stores work requests and results, and a Windows service that implements its business logic. These two elements are typically implemented on separate servers to provide a fault-resilient solution using a clustered control database and multiple FastStats services.

• Core i5 or better
• 8GB RAM or more
• Minimum 1 GB required for each FastStats Service
• 64-bit version Windows Server 2016 Standard or later with latest service packs and updates
• Microsoft .NET 4.8 (minimum) with the latest version recommended.

Database server¶

SQL Server Database (one required). You can implement it on the same server as the Application Tier below. A second server is optional, to support a database cluster for fault resilience:

• Core i5 or better
• 8 GB of RAM or preferably more
• Large SQL data and log store partition (e.g. 500Gb RAID 5 or bigger). You may need additional storage, depending on your system.

Note that PeopleStage stores significant volumes of data for automated campaigns. Please use the FastStats System Size Ready Reckoner to estimate the space required.

This storage isn't for the main FastStats data. Instead, it holds the requests and responses for all users across all systems, and data for the Cascade or PeopleStage campaign management module. The volume of communication activity most likely determines the space required, if you're using the campaign management modules. Otherwise, the number of user requests determines it.

• 100Mbps network card
• Backup device
• Windows Server 2016 Standard or later with latest service packs and updates
• SQL Server 2016 SP2 or later
• You can licence SQL Server using either a Processor or CAL (Client Access Licence) model. The CAL approach is appropriate if:
  • You aren't using the FastStats API
  • You can count the number of users that will access the system
  • The number of users you have doesn't mean a CAL-based licence model is more expensive than a Processor-based one.

Otherwise, use the Processor model.

See http://go.microsoft.com/fwlink/?LinkId=230678 for more information.

• Enterprise Edition for Clustered operation for fault resilience
• For PeopleStage, Apteco recommend SQL 2016 or later
• Higher RAM specified as all usable by SQL Server.
• The Apteco Marketing Suite uses SQL Full Recovery model by default, so you should budget for regular database and log backups.
• Access to SMTP Server for email to perform scheduled task reports

Database tier¶

The Database Tier consists of a database build server and a production server that operates the database using a Windows Service.

FastStats build server¶

You need one build server, which you can implement on the same server as the business tier / FastStats service. Use multiple servers if building multiple databases:

• Core i5 or better. Multiple cores improve performance for parallel data extract, parallel sorting, etc.
• 8 GB of RAM
• Large FastStats database store partition (e.g. 2Tb RAID 5). Size depends on the number and size of the FastStats databases it supports.
• Use the FastStats System Size Ready Reckoner spreadsheet to create an accurate system size and workspace estimate.
• Windows Server 2019 or later with latest service packs and updates
• Microsoft .NET Framework v 4.8 minimum, latest version recommended.
• 100Mbps or higher network card
• Backup device

The FastStats Server database engine, which powers the counting and analysis functions, can run on Windows Server 2016 or later. However, Apteco recommend Windows Server 2019 or later.

FastStats production server¶

You need one production server, which you can implement on the same server as the business tier described earlier. A second server is optional for fault resilience, and further servers are optional for additional capacity:

• Core i5 or better
• RAM dependent on number of systems envisaged.
• For optimum performance, aim for at least 2Gb for each FastStats database the server supports.
• Large FastStats database store partition (e.g. 2Tb RAID 5). Size depends on the number and size of the FastStats databases it supports. You can estimate this once you've created a prototype design.
• Windows Server 2019 or later with latest service packs and updates
• Microsoft .NET Framework v4.6.2 minimum, latest version recommended
• 100Mbps or higher network card
• Backup device

General notes on hardware specification¶

• If possible within the budget, you should choose high-speed disk drives (i.e. 15,000RPM devices). Solid-State Drive (SSD) devices operate fine and may give a performance improvement through reduced latency and improved read rates (FastStats is primarily limited by processor capacity and read speed).
• Apteco recommend Windows Server 2019 for any new installations.
• For larger FastStats databases, provide additional storage space. For example, to support automated deployment from FastStats Designer, the production server needs space for the following major elements:
• 1 copy of the deployment zip, approximately 10% the size of the FastStats system
• 3 copies of the FastStats database (new, current, and archive), approximately the same size as the raw data, though often smaller. The archive automatically zips after deployment.
• User's exported data files (size depends on the amount of data exported and how long you keep the files; provide 100MB per user).
• See the FastStats System Size Ready Reckoner spreadsheet available from Apteco
• The Cascade module requires additional SQL database storage space for the communication history table. This may run to hundreds of millions of rows, each containing approximately 50 bytes (i.e. a few tens of gigabytes once you account for fragmentation, logs, and indexes).
• The PeopleStage database requires significant storage space: provide 500 bytes per communication. The database can therefore grow to hundreds of gigabytes in a typical high-use implementation.
• For additional capacity choose to use multi-core, multi-processor servers.
• For additional resilience and read speed, you may choose to use RAID 5 drive storage.
• For performance and to remove replicated copies of the FastStats database, you can use a Storage Area Network (SAN) for the large FastStats database store partition.
• Note that there's no need to attempt to choose a server that will provide for all future capacity requirements. The FastStats.NET architecture supports the expansion of capacity through the addition of FastStats Servers in the Database Tier. The individual machines in the implementation can be of moderate specification, but used together to produce massive capacity.

Contact Apteco for specific consultation on a particular implementation.

User machine requirements¶

The following minimum specifications apply to machines running Apteco client applications.

Apteco FastStats¶

Apteco FastStats users' machines require Microsoft .NET Framework v 4.8 or later.

The processing speed and capacity of the Apteco FastStats interface will depend on the power and capacity of the users' machines. FastStats can use large amounts of RAM to deliver powerful results, such as multi-dimensional cubes or large profile reports. It may not operate on users' machines with insufficient memory.

Apteco recommend:

Users' client machines don't need significant bandwidth to connect and use the system. The initial application download will be approximately 20MB, but the system manages this download process and displays a progress bar during the download. Once downloaded, the application remains on the user's PC. It only needs re-downloading when the software version on the server changes.

Normal use of the application doesn't require significant bandwidth, but file transfers may cause higher network use.

Apteco PeopleStage¶

Apteco PeopleStage users' machines require Microsoft .NET Framework v 4.8 or later.

The speed of the Apteco PeopleStage interface will depend on the power and capacity of the users' machines. PeopleStage makes extensive use of graphics transforms and works best with an accelerated video card. Windows.NET includes a mechanism for ranking graphics performance. FastStats PeopleStage requires tier 2: contact Apteco for a test application that determines the graphics tier for existing machines.

Apteco recommend:

Users' machines don't need significant bandwidth to connect and use the system. The initial application download will be approximately 13MB, but the system manages this download process and displays a progress bar during the download. Once downloaded, the application remains on the user's PC. It only needs re-downloading when the software version on the server changes. Normal use of the application doesn't require significant bandwidth.