A confluence of factors, including technology trends and emerging applications, are driving greater emphasis on accelerator-based computing. Fully realizing the benefit of these accelerators often requires tight integration with their memory systems. However, in many high-performance computing systems, general-purpose “host” processors continue to play an important role. As such, moving data between the host’s memory space and the memory associated with accelerators introduces an additional bottleneck. One approach to addressing this bottleneck is to use the memory that is tightly integrated with the accelerators as the host’s main memory as well, essentially leading to the accelerators being viewed as near memory computing units from the host’s perspective. Realizing such an organization, however, requires careful attention to factors such as the ability to efficiently incorporate multiple memory modules with near-memory accelerators and balancing conflicting memory performance requirements of the accelerators and the host. This talk will discuss some of these practical challenges associated with near-memory accelerators in high-performance systems and some preliminary directions for solutions.